CN114676138A - Data processing method, electronic device and readable storage medium - Google Patents

Abstract

The present application relates to the field of computer technology, and in particular, to a data processing method, an electronic device, and a readable storage medium. The data processing method includes: receiving a data query request; determining a target data set corresponding to the query request, and determining at least one data subset of the target data set and an index information set corresponding to each data subset; The target index information matching the data query request and the target data entity corresponding to the target index information are determined in the first information and the second information of each index information set. The data processing method in the embodiment of the present application increases the storage and query of string type data, and enriches the query capability of data entities.

Description

Data processing method, electronic device and readable storage medium

technical field

The present application relates to the field of computer technologies, and in particular, to a data processing method, an electronic device, and a readable storage medium.

Background technique

With the development of computer technology, the amount of data is increasing, and the difficulty of data processing is also increasing. The data includes structured data and unstructured data. Structured data is generally represented as a two-dimensional table structure, and structured data can be, for example, numbers, dates, string data, and the like. Unstructured data is generally processed as a vector, and unstructured data can be, for example, pictures, videos, text data, and so on.

For unstructured data, currently, computers generally process it as vector type data. When storing data for a data entity that contains both structured data and unstructured data, the storage area is divided into non-string type scalars. Columns and vector columns are stored, or the storage area is divided into non-string scalar rows or vector rows for persistent storage, and an index of vector columns is constructed. When data query is required, the filter conditions about the vector column will be generated according to the query request, and the vector column data will be filtered according to the index of the constructed vector column. For example, for a face image, the unstructured data of the vector type representing the face data is obtained after data processing, and then combined with the structured data of the scalar type of the age of the face image, it is stored as a vector representing the face data. column and a scalar column representing the age in years, where the face image and its corresponding age can be used as a data entity.

The above data processing method, when querying a data entity, only supports the query of non-string type scalar type data and vector type data, the queryable data types are few, and the data query efficiency is low, thereby making the computer query the data entity. Ability is not high.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a data processing method, an electronic device, and a readable storage medium, which increase the storage and query of string type data, and enrich the query capability for data entities.

In a first aspect, an embodiment of the present application provides a data processing method for an electronic device, including:

Receive data query requests;

determining a target data set corresponding to the query request, and determining at least one data subset of the target data set and an index information set corresponding to each data subset, wherein the data subset includes a plurality of data entities, and Some data entities in the plurality of data entities include first data of string type and second data of non-string type; and the index information set includes at least the first data used to characterize the data subset The first information of the corresponding relationship with the string index, and the second information used to characterize the corresponding relationship between the string index in the data subset and the data entity;

The target index information matching the data query request and the target data entity corresponding to the target index information are determined from the first information and the second information of the multiple index information sets of the target data set.

The data processing method provided by the embodiments of the present application can query the string data of the unstructured data by querying the target data set including the string data, and expand the data types that can be queried. At the same time, the query of string data and non-string data can be carried out at the same time, which enriches the query ability of data entities.

In a possible implementation manner of the above-mentioned first aspect, the first information used to represent the correspondence between the first data in the data subset and the character string index is dictionary tree information.

In a possible implementation manner of the above-mentioned first aspect, the data query request includes third data of string type;

Determining, in the first information and the second information of the multiple index information sets of the target data set, the target index information that matches the data query request and the target data entity corresponding to the target index information ,include:

determining state information of the at least one data subset of the target data set, the state information including an archive state and a growth state;

If the at least one data subset is in the archived state, determining the target string index corresponding to the third data as the target index information according to the dictionary tree information;

According to the second relationship, the target data entity corresponding to the first target string index is determined.

In a possible implementation manner of the above-mentioned first aspect, determining the target string index corresponding to the third data as the target index information according to the dictionary tree information includes:

looking up the third data in the dictionary tree information;

It is determined according to the dictionary tree information that the target character string index corresponding to the third data is the target index information.

In a possible implementation manner of the above-mentioned first aspect, the first information and the second information of the multiple index information sets of the target data set are determined to match the data query request. The target index information and the target data entity corresponding to the target index information also include:

It is determined that the target data entity is the data query result corresponding to the data query request.

In a possible implementation manner of the above-mentioned first aspect, the data query request further includes fourth data of a non-string type.

In a possible implementation manner of the above-mentioned first aspect, the data query request includes a query condition, and the query condition includes at least one of the following:

boolean expression;

Prefix matching conditions that characterize string prefixes;

An exact match condition that characterizes a string.

In a second aspect, an embodiment of the present application provides a data processing method for an electronic device, characterized in that it includes:

receiving a data insertion request, where the data insertion request includes a character string set to be inserted, and the character string set includes a plurality of character string data;

In response to the data insertion request, write the plurality of string data into corresponding data subsets of the target data set respectively, each data subset includes at least one data entity, and the at least one data entity includes at least one data entity. A string of data.

It can be understood that the data processing method provided by the embodiment of the present application can quickly and effectively realize the construction of the index of the string data by constructing the corresponding dictionary tree information and mapping information for the string data of each data subset. The mapping relationship can be used to implement the construction of indexes on the field columns of non-string data, providing a basis for more types of scalar indexes.

In a possible implementation manner of the above-mentioned second aspect, the above-mentioned method further includes:

obtaining the at least one character string data in the data subset;

constructing, according to the at least one character string data, third information of the data subset for representing the correspondence between the at least one character string data in the data subset and a character string index;

According to each character string data and the corresponding character string index in the first information, and the data entity corresponding to each character string data, construct the character string index and the data characterizing the data subset. The fourth information of the corresponding relationship of the entity.

In a possible implementation manner of the above second aspect, the writing of the plurality of character string data into the corresponding data subsets in response to the data insertion request includes:

In response to the data insertion request, dividing the plurality of character string data into M character string subsets, and sending the character string subsets to corresponding M data nodes, where M is greater than or equal to 2;

Writing each character string data in the character string subsets of the M data nodes into the corresponding data subsets respectively.

In a third aspect, embodiments of the present application provide an electronic device, including: one or more processors; one or more memories; one or more memories stored with one or more programs, when the one or more programs are When executed by one or more processors, the electronic device is made to execute the data processing method of the first aspect or the second aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, where instructions are stored on the storage medium, and when the instructions are executed on a computer, the instructions cause the computer to execute the data processing method of the first aspect or the second aspect.

In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program/instruction, which implements the data processing method of the first aspect or the second aspect when the computer program/instruction is executed by a processor.

Description of drawings

FIG. 1 is a schematic diagram of a data query system according to some embodiments of the present application;

Fig. 2 shows a schematic structural diagram of a data table exemplarily according to some embodiments of the present application;

FIG. 3 is a schematic flowchart illustrating a data processing method according to some embodiments of the present application;

FIG. 4 is a schematic flowchart illustrating a data insertion according to some embodiments of the present application;

FIG. 5 is a schematic flowchart illustrating a data insertion according to some embodiments of the present application;

FIG. 6 is a schematic flow chart illustrating an example of constructing an index according to some embodiments of the present application;

FIG. 7 is a schematic flow chart illustrating an example of constructing an index according to some embodiments of the present application;

FIG. 8 is a schematic flowchart illustrating a data processing method according to some embodiments of the present application;

FIG. 9 is a schematic diagram of an interaction process exemplarily showing a data query process according to some embodiments of the present application;

FIG. 10 is a schematic diagram of an interaction process exemplarily showing another data query process according to some embodiments of the present application;

FIG. 11 is a schematic diagram illustrating a hardware structure of an electronic device according to some embodiments of the present application.

Detailed ways

The present invention is described below based on examples, but the present invention is not limited to these examples only. In the following detailed description of the invention, some specific details are described in detail. The present invention can be fully understood by those skilled in the art without the description of these detailed parts. Well-known methods, procedures, procedures, components and circuits have not been described in detail in order to avoid obscuring the essence of the present invention.

Furthermore, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.

Unless clearly required by the context, words "including", "comprising" and the like throughout this specification should be construed in an inclusive rather than an exclusive or exhaustive sense; that is, in the sense of "including but not limited to".

In the description of the present invention, it should be understood that the terms "first", "second" and the like are used for descriptive purposes only, and should not be construed as indicating or implying relative importance. Also, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

Before introducing the solutions in this application, in order to facilitate understanding of the solutions in this application, some concepts and terms involved in this application are introduced below with reference to FIG. 1 and FIG. 2 .

FIG. 1 shows a data query system according to an embodiment of the present application.

As shown in FIG. 1 , the data query system includes an access component 1 , a coordination service component 2 , a message storage component 3 , an execution component 4 and an object storage module 5 .

Among them, the proxy component 1 is used to receive external data insertion/deletion/query and other requests, and return data processing results to the outside. The proxy component 1 includes a plurality of proxy modules, such as the proxy module 11 and the proxy module 12 in FIG. 1 .

The coordination service component 2 is used for allocating data processing tasks to each module of the execution component 4, and saving the status information of each module and so on. The coordination service component 2 may include a coordination management module 21 , a coordination query module 22 , a coordination data module 23 , a coordination index module 24 , and a metadata storage module 25 that stores information such as the status of the execution component 4 .

The message storage component 3 is used to receive data manipulation language (Data Manipulation Language, DML) commands corresponding to requests for data addition/insertion/deletion/query sent by the proxy component 1, and temporarily store them in the message storage component 3 in the form of logs in the log storage module 31.

The execution component 4 is used to execute the data processing task assigned by the coordination service component 2 and the DML command initiated by the proxy module of the proxy component 1, and write the data stored in the log storage module 31 into the object storage module in the form of a binary log (binlog). 5 in. The Zhixingao component 4 is also used to obtain the binlog file stored in the object storage module, build an index for the data in the binlog file, and store it in the object storage module 5 .

The object storage module 5 is used to store the binlog file and the index constructed by the execution component 4 . In some embodiments, the log file 51 in the object storage module 5 may store a binlog file, and the index file 53 in the object storage module 5 may store an index of a scalar/vector, wherein the scalar includes numeric data, string data data, date data, and boolean data.

The data stored in the object storage module 5 will be introduced below with reference to FIG. 2 . FIG. 2 is a schematic structural diagram of a data table in an embodiment of the present application.

As shown in FIG. 2 , in some embodiments, the data in the object storage module 5 is stored as a data table 6 . The data table 6 includes a plurality of data partitions, for example, a data partition 601, a data partition 602, and the like. Each data partition may include multiple data segments, for example, data partition 601 includes data segment 1 and data segment 2, and data partition 602 includes data segment 3, data segment 4, and data segment 5. Each data segment includes a*b data fields, where each row of data fields in row a is a data entity, that is, corresponding to the same unstructured data, and each column of data in column b is of the same data type . Wherein, each data partition can divide the data according to the rules defined by the user, for example, multiple data partitions are divided according to the date, and multiple data partitions are divided according to the user's location.

For example, taking the unstructured data as a face image as an example, the computer-readable data corresponding to a face image is a data entity. The data entity corresponding to each face image may include an integer column field composed of age, a vector column field composed of face data, and a string column field representing the label of the face data, and the computer-readable data corresponding to the four face images Stored as a set, where the integer column field consisting of age and the string column field representing the label of the face data are scalar column fields. Then, the added 10 face images can be represented as 3 data segments in the data table 6, that is, each data segment can include 4*3 data fields. The label of the face data may be, for example, the name of the person corresponding to the face image.

In some embodiments, each data segment includes a*b data fields, wherein each column of data fields in column b is a data entity, that is, corresponding to the same unstructured data, each column in row a is a data entity. The data are of the same data type.

Continuing to refer to FIG. 1 , in some embodiments, the execution component 4 includes at least one query module 41 , at least one data module 42 and at least one index module 43 .

The data module 42 is configured to receive the data stored in the log storage module 31, and write the data therein in the form of a binary log (binlog) into the object storage module 5 (ie, the log file 51). If the data is stored in different field columns according to different data types, the log file can include scalar columns and vector columns.

The query module 41 is configured to receive the query operation data stored in the log storage module 31, and simultaneously load the log file 51 in the object storage module 5 into the memory of the query module 41, and then complete the data query based on the query operation data. The query module 41 can realize the functions of search, mixed search and query.

It can be understood that the search means performing a nearest neighbor search on the vector column field, and returning the k data entities with the highest matching degree with the query condition. The hybrid search means to search for at least two data fields including the vector column field in the string type scalar column field, the vector column field and the non-string type scalar column field respectively according to a preset search order. Query means to filter non-string type scalar column fields and string type scalar column fields, and return the data entities corresponding to non-string type scalar column fields or string column fields that meet the query conditions.

For example, suppose that each data entity represents an image of a face, and each data entity includes an integer column field representing age, a vector column field representing face data, and a string column field representing name. The search can be to find the face image most similar to the target face. At this time, the face data can be searched for the nearest neighbors, and the data entities corresponding to the k face data most similar to the target face can be returned. The hybrid search can query the face image with the name A that is most similar to the target face. At this time, you can filter the string column corresponding to the name according to the query conditions to get the name A, and then filter the results. Participate in the nearest neighbor search of face data, and return k face images named A and most similar to the target face in the nearest neighbor search results. The query can be to query the first 5 columns of data in the data table 6. In this case, the data fragments corresponding to the first 5 columns of data in the data table 6 can be filtered according to the query conditions (each data fragment includes 3 columns of data fields), and then The face image corresponding to the filtered data field will be returned.

As mentioned above in the background art, the existing data processing methods have low query capabilities for unstructured data. In order to solve the above problems, the present application discloses a data processing method, which can be applied to a data query system. And the data table in the data query system stores the string data and the index information set corresponding to the string data of each data segment. Specifically, the method includes: receiving a data query request; determining a target data set corresponding to the query request, and determining at least one data subset of the target data set and an index information set corresponding to each data subset, wherein the data subset includes a plurality of data entities, and some of the data entities in the plurality of data entities include first data of a string type and second data of a non-string type; and the index information set includes at least the first data used to characterize the data subset The first information of the corresponding relationship with the string index, and the second information used to characterize the corresponding relationship between the string index in the data subset and the data entity; the first information and the first information in the multiple index information sets of the target data set In the second information, the target index information that matches the data query request and the target data entity corresponding to the target index information are determined.

In this embodiment of the present application, the data query request may be a request received from the outside, and the request may be used to query a data entity, where the data entity includes at least one of structured data and unstructured data, wherein the structured data storage In the database, a two-dimensional table structure can be used to logically express the implemented data, and each data has a specific meaning, such as integer data, string data, and so on. Unstructured data refers to data whose data structure is irregular, there is no unified predefined data model, and it is inconvenient to use the two-dimensional logical table of the database to represent the data. Unstructured data includes pictures, video, audio, natural language, etc., accounting for 80% of all data. The processing of unstructured data can be transformed into vector data for processing by various artificial intelligence (AI) or machine learning (ML) models.

It can be understood that, in some embodiments, the target data set is the data table 6 in FIG. 2 , the data subsets are the data segments in FIG. 2 , such as data segment 1 to data segment 5, each data entity corresponds to an unstructured The data, the first data and the second data may be data fields in the data segment, and the first data and the second data are stored in the form of binlog.

It can be understood that the second data of the non-string type may be a non-string type scalar field, such as an integer field, a floating-point field, or a vector field, such as a floating-point vector field, etc., which is not limited in this application .

It can be understood that the index information set includes index information of string type data (hereinafter referred to as string data) and index information of non-string type data.

In the data processing method provided by the embodiment of the present application, the storage of character string data is added to the target data set, and first information representing the correspondence between the first data in the data subset and the character string index is constructed corresponding to the character string data. Furthermore, when querying a data entity, it can not only query vector data and non-string scalar data, but also query string data based on the first information, which expands the queryable data types of the data entity. . At the same time, the query of string data and non-string data can be carried out at the same time, which enriches the query ability of unstructured data.

In addition, the index information set in this embodiment of the present application further includes second information used to represent the correspondence between the string index in the data subset and the data entity, and based on this second information, a query for a scalar string field column can be implemented , to implement index building for scalar type data, that is, the embodiments of the present application can provide support for scalar indexes, and provide a basis for the construction of more types of scalar indexes.

In order to support the query of character string data, the data processing method provided in the embodiment of the present application needs to complete the storage of character string data and the construction of an index first. The following describes the data storage process and the index construction process in the embodiment of the present application with reference to FIG. 3 .

FIG. 3 is a schematic flowchart of data storage and index building in an embodiment of the present application.

As shown in Figure 3, the process of data storage and recipient indexing includes:

301: A data insertion request is received. The data insertion request includes a character string set to be inserted, and the character string set includes multiple character string data. The string set to be inserted is string-type scalar data.

It can be understood that when the proxy module in the data query system receives the data insertion request, it will generate the data insertion command and the data to be inserted according to the data insertion request, and store the data insertion command and the data to be inserted in the log storage module in the form of logs. 31.

In some embodiments, the data insertion request may further include non-string data, such as non-string type scalar data, vector data, and the like. For example, if the data insertion request is to insert a face image, the data insertion request may include integer scalar data representing age, vector data representing face data, and character string data representing name.

302: In response to the data insertion request, write a plurality of string data into corresponding data subsets respectively. Wherein, each data subset includes at least one data entity, and at least one data entity includes at least one character string data.

It can be understood that the data subset includes at least one data entity corresponding to the unstructured data, and each data entity includes a string data as a string field of a data entity. It can be understood that after the data module 42 in the data query system receives the data insertion task generated by the coordination service component 2, the data module 42 will read the data to be inserted from the log storage module 31, and insert the data to be inserted and the data into Operations are stored in the object storage module 5 in the form of binlogs.

The storage space occupied by each data subset in the embodiment of the present application is preset. In some embodiments, if the data entity corresponding to the string data to be inserted does not exist in the data table, the plurality of string data are respectively written into the corresponding data subsets, that is, starting from the data subset that can currently store data , and sequentially write multiple string data into the string column in the corresponding data entity in the data subset. After the string data storage of the current data subset reaches the upper limit, write the rest of the string data to be stored into the next data. subset. For example, taking the storage limit of the data subset as 4 data entities and 10 images to be inserted as an example, if the data entities corresponding to the 10 images to be inserted do not exist in the data table 6, it is determined that the first data entity that has not reached the storage upper limit is determined. The position of a data subset, and then write the string data of 10 images into the data subset in turn, until the string data of the data subset reaches the storage limit of 4, and then start to write the data of the next data subset. .

In other embodiments, if the data entity corresponding to the string data to be inserted already exists in the data table, then multiple string data are written into the corresponding data subsets, that is, the query module 41 performs data query, and determines The position of the data subset corresponding to the data entity in the data table 6, and then insert a plurality of string data into the string column in the corresponding data entity in sequence. For example, taking the storage limit of the data subset as 4 data entities and 10 images to be inserted as an example, if the data entities corresponding to the 10 images to be inserted already exist in the data table 6, the data entities of the 10 images are determined. position, and then sequentially write the string data of 10 images into the string column in the corresponding data entity.

In some embodiments, the storage space occupied by different data subsets may be the same or different, which is not limited in this application.

In some embodiments, each data entity may further include non-string data, as a non-string field of the data entity, such as non-string scalar fields and vectors including integer scalar fields, floating-point scalar fields, etc. field.

In some embodiments, 302 may specifically include: in response to the data insertion request, dividing the plurality of string data into M string subsets, and sending the string subsets to the corresponding M data nodes, where M is greater than Equal to 2; each character string data in the character string subsets of the M data nodes is written into the corresponding data subset respectively.

It can be understood that the data node is the data module 42 in FIG. 1 . In the embodiment of the present application, the multiple character string data is divided into M small batches of data, that is, into M character string subsets, and then when data storage is performed, the M character string subsets can be simultaneously sent to the corresponding Data module 42 and complete data storage. 302 will be further described below in conjunction with FIG. 4 and FIG. 5 .

FIG. 4 and FIG. 5 are flowcharts of a data storage process in an embodiment of the present application.

As shown in FIG. 4 , in some embodiments, the data to be inserted includes IDs of data entities, string data, and vector data. The ID in it can also be understood as the primary key hash value of the data entity.

Specifically, the proxy module divides the data entity into two small batches of data (batch data 1 and batch data 2) according to the primary key hash value of the data entity, and stores them in the corresponding data nodes (nodes) in the log storage module 31 s1 and node s2), each data node can be served by a query module or data module. The query module 411 and the data module 421 corresponding to the node s1, and the query module 412 and the data module 422 corresponding to the node s2.

The data module 421 can obtain the batch data 1 stored on the node s1, the data module 422 can obtain the batch data 2 stored on the node s2, and the data module 421 and the data module 422 can simultaneously store the batch data 1 and batch data 2 persistently to Object storage module 5.

As shown in FIG. 5 , in some embodiments, data of different nodes may be transmitted to the data module through different data transmission channels for persistent storage of the data.

For example, the batch data 1 of the node s1 may be transmitted to the data module 421 through the channel A, and the data module 421 writes the batch data 1 into the corresponding multiple data segments 61 . At the same time, the batch data 2 of the node s2 can be transmitted to the data module 422 through the channel B, and the data module 422 writes the batch data 2 into the corresponding multiple data segments 61 .

It can be understood that each row of data in the data segment 61 represents a data entity, that is, a corresponding piece of unstructured data. Each data entity may include multiple fields, such as an id field (integer scalar field), a string field (string scalar field), and a vector field. When the data size of one data segment reaches the storage limit, the data entity will be written into the next data segment 61 . The storage upper limit may be, for example, 512MB.

303: Acquire at least one character string data in the data subset.

It can be understood that obtaining at least one character string of data bytes means obtaining character string column data in the data subset. In some embodiments, the retrieved string data may include a string field and a row offset of the string field relative to the first row data entity of each data subset.

For example, if the data subset includes the string field column S as shown in FIG. 6 , the acquired character string data includes the string field column S and its corresponding row offset list R. where the row offset list consists of the row offset of each string field in the string field column S relative to the string field of the first row of the data subset.

It can be understood that the indexing module 43 loads at least one character string data of the data subset from the object storage module 5 and stores it in the memory of the indexing module 43 .

304 : According to the at least one character string data, construct a data subset to represent third information of the correspondence between the at least one character string data in the data subset and the character string index.

In some embodiments, the third information is dictionary tree information. It can be understood that the indexing module 43 constructs the dictionary tree information according to the character string data of the data subset loaded in the memory.

It can be understood that the dictionary tree information is to regard each string field in the data subset as a character sequence, and then construct a top-to-bottom tree structure according to the sequence of the character sequence. Each edge in the tree structure corresponds to a character, and each child node of the dictionary tree can be represented as an index of a character sequence consisting of characters from the root node to the child node.

For example, as shown in FIG. 6, for the string field column S, a set composed of string fields (a, ab, abc, ac, acd, acc, accd, acdd) is included. The index module 43 uses the character "a" as the root node, the remaining characters in the data subset as edges, and the tree ID (ie, the string index) as the child node, and constructs a dictionary tree T (ie, the upper dictionary tree information in the text). The edges of the dictionary tree T include (c, b, c, b, dd, c, d), and the tree IDs include 0 to 7. The character sequence composed of characters stored in the edge connecting the root node and any child node in the dictionary tree T represents a string field in the string field column S.

305: According to each character string data in the first information and the corresponding character string index, and the data entity corresponding to each character string data, construct a data subset for characterizing the character string index in the data subset and the corresponding character string index. The fourth information describing the corresponding relationship of the data entities.

It can be understood that, in some embodiments, the indexing module 43 constructs the index and the fourth information of the data entity according to the constructed dictionary tree information and the row offset of each string field relative to the first row of data entities.

In some embodiments, the fourth information (hereinafter referred to as a mapping relationship) may be characterized as a corresponding relationship between tree IDs and data entities in the dictionary tree information. For example, the mapping relationship can be represented as the index ID in FIG. 6 , and the index ID is equal to the number of rows of the string field S. Each row of data in the index ID represents the tree ID of the string field of the row in the dictionary tree T, and the row where the tree ID is located is relative to the row offset of the first tree ID in the index ID, indicating the corresponding tree ID. The row offset of the string field relative to the first string field in the string field column.

In some embodiments, after the data module 42 performs the above steps 301 and 302 to complete the writing of a data subset, the index module 43 can perform steps 303 to 305, and the data module 42 can complete the writing of the next data subset. data write.

The index construction process of the data subset will be further introduced below with reference to FIG. 7 .

FIG. 7 shows a flowchart of constructing an index of character string data in an embodiment of the present application.

In some embodiments, as shown in FIG. 7 , the object storage module 5 stores a data segment 611 and a data segment 612 . The data segment 611 includes multiple data entities, and each data entity includes two field columns: field column 1 and field column 2. The fields in each field column are stored as data fields in the form of binlog, for example, in field column 1 of log 1 and log 2, field column 2 of log 3 and log 4. The data segment 612 includes multiple data entities, and each data entity includes two field columns: field column 3 and field column 4. The fields in each field column are stored as data fields in the form of binlog, for example, in field column 3 of log 5, field column 4 of log 6.

The indexing module 43 loads the string field columns in the data segment 611 and the data segment 612 (assuming that the field column 2 and the field column 4 are string field columns) into the memory of the indexing module 43, and corresponds to the data segment 611 and the data segment. The string field column of 612 performs build index task 1 and build index task 2. Specifically, the indexing module 43 performs index building task 1, builds an index for field column 2, and generates dictionary tree information 5311 and mapping information 5321. The indexing module 43 performs index building task 2, builds an index for field column 4, and generates dictionary tree information 5312 and mapping information 5322.

After the indexing module 43 completes the index construction, the generated dictionary tree information and mapping information are respectively stored in the index file 53 of the data segment corresponding to the object storage module 5 . Specifically, the dictionary tree information 5311 and the mapping information 5321 are stored in the index file corresponding to the data segment 611 in the object storage module 5, and the dictionary tree information 5312 and the mapping information 5322 are stored in the data segment 612 in the object storage module 5. in the corresponding index file.

It can be understood that in the embodiment of the present application, corresponding dictionary tree information and mapping information are all constructed for each data subset. ) Core(TM) i7-8700 CPU@3.20GHz, the machine configuration of the central processing unit with 32G storage space as an example, the time and space overhead of index construction of string data with different data volumes are tested, and the test results are as follows shown in Table 1.

Table 1

It can be seen from Table 1 above that when the number of lines of the string remains unchanged and the length of the string continues to increase, the time to build the index, the time to load the dictionary tree, the amount of index data, the amount of string data, and the prefix query time are all large. The magnitude increased. When the number of string rows increases and the string length increases, the index construction time, the amount of quoted data, the amount of string data, and the prefix query time all increase significantly. At the same time, when the number of rows is too large or the string is too long, the index construction may fail.

It can be understood that as the length and number of rows of string data increase, the time and space overhead of string indexing increases significantly, and the index may also fail to be constructed.

In the embodiment of the present application, by dividing the target data set into multiple data subsets, and then constructing dictionary tree information and mapping information respectively for the character string fields in each data subset, the construction time of the index information set can be shortened, and the time required to construct the index information set can be shortened. The memory space occupied by each dictionary tree information, while reducing the possibility of index construction failure.

In some embodiments, the indexing module 43 may also acquire vector field columns in the data subset, and build an index for the vector field columns according to a preset index type. The index type may be, for example, a flat lattice transformation (Flat LatticeTransformer, FLAT), an inverted product quantization transformation (Inverted File Product Quantization, IVF_PQ), and the like.

The data query process in the embodiment of the present application will be further introduced below with reference to FIG. 8 .

FIG. 8 shows a flowchart of a data query in an embodiment of the present application.

As shown in FIG. 8, in some embodiments, the process of data query includes:

801: Receive a data query request.

It can be understood that after the proxy module receives an external data query request, it generates a data query command and data query conditions according to the data query request, and the coordination processing component 2 can generate a corresponding data query task. The proxy module stores the data query commands and data query conditions in the log storage module 31 in the form of logs. The query module 41 receives the data query task generated by the coordination processing component 2 , and obtains data query conditions from the log storage module 31 .

In some embodiments, the data query conditions include query of string data. For example, the data query condition in the data query request is to query the face image whose name is A. In other embodiments, the data query condition may also include a query of non-string data, such as non-string scalar data, vector data, etc. including integer scalar fields, floating-point scalar fields, and the like. For example, the data query request is to query the face data closest to the target face data.

802: Determine a target data set corresponding to the query request, and determine at least one data subset of the target data set and an index information set corresponding to each data subset.

The data subset includes multiple data entities, and some of the multiple data entities include first data of a string type and second data of a non-string type. In addition, the index information set includes at least first information for characterizing the correspondence between the first data in the data subset and the string index, and first information for characterizing the correspondence between the character string index and the data entity in the data subset. Second information (ie, the mapping information in the foregoing, hereinafter referred to as the mapping information). In some embodiments, the first information may be characterized as dictionary tree information.

It can be understood that when the query module 41 receives the data query task, it can load the data subset of the target data set from the object storage module 5 into the memory of the query module 41 .

In some embodiments, the subset of data includes two states, a growth state and a sequestered state. Furthermore, the data subsets loaded into its memory by the query module 41 include two types: growth data subsets and archived data subsets. The growth data subset can continuously write data to it, and when the data volume of the growth data subset exceeds the preset storage limit or no data is written within a preset time period, the growth data subset will be converted into a sealed data subset. For queries of growing data subsets, a violent search method can be used. The archived data subset does not support data writing, and can only realize data deletion, and the index module 43 obtains the string data in the archived data subset to construct an index, and the query module can use the data processing method provided in the embodiment of the present application. Query a subset of data.

In some embodiments, the data that query module 41 loads into its memory also includes a set of index information for a subset of the data. Further, the dictionary tree information in the index file 53 loaded by the query module 41 can support "==", "!=", ">", ">=", "<", "<=", prefix search, exact Find and other query conditions.

In some embodiments, the query condition of the query module 41 can be expressed as a Boolean expression, for example, the query condition is a string field>"abc". In some embodiments, the query condition can also be expressed as a specified string prefix, for example, the query condition is that the prefix of a string field is "abc". In some embodiments, the query condition of the query module may also be expressed as a filter condition of a non-string field column, for example, the query condition is to select the first 100 columns in the non-string type scalar field column.

In some embodiments, the index information set includes dictionary tree information and mapping information, wherein the dictionary tree information represents the correspondence between string field columns and string indexes in the data subset, and the mapping information represents the Correspondence between string indexes and data entities. For example, the dictionary tree information in the index information set may be represented as the dictionary tree T shown in FIG. 6 , and the mapping information may be represented as the index ID shown in FIG. 6 . Further, in some embodiments, the mapping relationship may also represent the corresponding relationship between the vector index corresponding to the vector field and the data entity.

803: Determine, from the first information and the second information of the multiple index information sets of the target data set, the target index information that matches the data query request and the target data entity corresponding to the target index information.

It can be understood that the query module 41 can determine the target index information that matches the data query condition in the index information set based on the loaded index information set.

In some embodiments, the data query request is a search, and the corresponding query condition is a query on a vector field. In some other embodiments, the data query request is a mixed search, and the corresponding query conditions include at least two of a string field column (ie, a string-type scalar column), a vector field column, and a non-string-type scalar field column. A query on a field column. In other embodiments, the data query request is a query, and the corresponding query condition is a query on a scalar field column (including a string field column and a non-string type scalar field column). The data query request and the query process will be further introduced below with reference to FIG. 9 and FIG. 10 .

In some embodiments, the data query request is a query for a string field column, and the query module 41 determines the target data entity corresponding to the target string index information according to the mapping information in the index information set. For example, the query module 41 filters out the tree ID that matches the query condition in the data query request from the dictionary tree information, and then the query module 41 characterizes the tree ID and the string field corresponding to the tree ID relative to the first tree ID in the data subset. The mapping relationship between the row offsets of each string field, determine the row offset of the string field corresponding to the filtered tree ID relative to the first string field in the data subset, and determine the data entities that meet the query conditions .

In some embodiments, if the data query request is a query for a vector field column, the query module 41 may determine the corresponding information of the target vector index according to the mapping information of the corresponding relationship between the vector index corresponding to the character field and the data entity in the index information the target data entity.

In some embodiments, the data entities that meet the query conditions queried by the query module 41 can be used as query results, and returned to the outside that initiates the data query request through the proxy module.

FIG. 9 is a schematic diagram of an interaction process of a data query provided by an embodiment of the present application.

As shown in FIG. 9 , in some embodiments, the data query request is a query, the query condition 901 includes string data, and the query module 41 may determine that the query condition 901 matches the query condition 901 according to the dictionary tree information 531 loaded from the object storage module 5 . The index list 902 composed of the string index of , and the index list 902 is the target index information. Further, the query module 41 generates a bitmap 903 representing whether each row of data in the data subset meets the query condition according to the mapping information 532 loaded from the object storage module 5 . In the bitmap 903, a value of 1 indicates that the string field corresponding to the value meets the query conditions, and the data entity corresponding to the value can be returned as the query result; a value of 0 indicates that the string field corresponding to the value does not meet the query conditions, The data entity corresponding to this value cannot be returned as a query result.

For example, the dictionary tree information in the index information set can be represented as the dictionary tree T shown in Fig. 6, the mapping information can be represented as the index ID shown in Fig. 6, and the query condition is the data whose string prefix is "ac", then the generated The index list 902 can be (1, 4, 3, 7, 5), and the bitmap 903 generated according to the mapping information 532 can be expressed as (0, 0, 0, 1, 1, 1, 1), that is, a string field column The data entities in S that meet the query conditions are the data entities in rows 4 to 8. The query module 41 can return the data entities in the fourth to eighth rows in the data subset to the outside that initiates the data query request through the proxy module.

In some embodiments, the data query request in FIG. 9 is a hybrid search, and the query condition 901 may also include vector data. After obtaining the bitmap 903, the query module 41 can select the string field corresponding to the value 1 to which it belongs. The vector field of the data entity participates in the neighbor query of vector data, and returns the data entities corresponding to the k vector fields that meet the query condition 901 .

It can be understood that the query conditions corresponding to the hybrid search may include at least two kinds of scalar data and vector data including vector data, which are not limited to the above examples, and are not described in detail in this application.

In other embodiments, the data query request is a search, the query condition 901 includes vector data, and the query module 41 may determine a vector index that matches the query condition through a neighbor search according to the index information of the vector data loaded from the object storage module 5 . The index list is composed, and the index list is the target index information. Further, the query module 41 generates a bitmap representing whether each row of data in the vector field column in the data subset meets the query condition according to the mapping information loaded from the object storage module 5 . In the bitmap, a value of 1 indicates that the vector field corresponding to this value meets the query conditions, and the data entity corresponding to this value can be returned as the query result; a value of 0 indicates that the vector field corresponding to this value does not meet the query conditions, and the value corresponds to The data entity cannot be returned as a query result.

For example, the data entity is a face image, the vector field represents the face data, and the query condition is to search for face data matching the target face data. The query module 41 obtains the index information of the face data in the data subset, and determines the k face data most similar to the target face data through the nearest neighbor search, and returns the face image corresponding to the k face data as the query result to the initiator. External to data query requests.

FIG. 10 is a schematic diagram of another interactive process of data query provided by an embodiment of the present application.

As shown in FIG. 10 , in some embodiments, the data query request is a query, and the data query request includes a query condition 1001 . The query condition 1001 includes non-string type scalar data, and the query module 41 can determine, according to the data subset loaded from the object storage module 5, the non-string type scalar field that matches the query condition 1001 relative to the first one in the data subset A list of row offsets 1002 consisting of row offsets for non-string scalar fields. Further, the query module 41 determines the string index corresponding to the row offset in the row offset list 1002 according to the mapping information 532 loaded from the object storage module 5, and determines the string 1003 corresponding to the string index according to the dictionary tree information , and returns the data entity to which the string 1003 belongs as the query result to the outside that initiates the data query request.

For example, the dictionary tree information in the index information set can be represented as the dictionary tree T shown in FIG. 6, the mapping information can be represented as the index ID shown in FIG. 6, and the query condition 1001 is non-string scalar data>100, then the query module 41 First, use the query condition 1001 to filter the non-string type scalar field column of the data subset, and filter out the row offset list 1002 of the non-string type scalar field that meets the query condition 1001 . Then the query module 41 can determine the string index corresponding to the row offset list based on the mapping information loaded from the object storage module 5, and then determine the string 1003 corresponding to the string index based on the dictionary tree information loaded from the object storage module 5. . The query module 41 can return the data entities corresponding to the character string 1003 in the data subset that meet the query condition 1001 to the outside that initiates the data query request through the proxy module.

It can be understood that the query module 41 maps the query corresponding to the non-string type scalar field column to the query on the string index of the string field, and then obtains the string 1003 corresponding to the string index based on the dictionary tree information. The mapping relationship can also be understood as the index of the string-type scalar field column, which can realize the query for the string-type scalar field column.

11 shows a schematic structural diagram of an electronic device 1100 according to an embodiment of the present application. The electronic device 1100 may include a processor 1110 , an internal memory 1120 , an interface module 1130 , a power module 1140 , and a wireless communication module 1150 .

It will be appreciated that the electronic device 1100 includes, but is not limited to, cell phones (including foldable cell phones), tablet computers, laptop computers, desktop computers, servers, wearable devices, head-mounted displays, mobile email devices, in-vehicle devices, portable devices Various electronic devices such as game consoles, portable music players, reader devices, televisions with one or more processors embedded in or coupled to them.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 1100 . In other embodiments of the present application, the electronic device 1100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 1110 may include one or more processing units, for example, the processor 1110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor, etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

A memory may also be provided in the processor 1110 for storing instructions and data. In the embodiment of the present application, the processor 1110 may execute the data processing method in the present application.

Internal memory 1120 may be used to store computer executable program code, which includes instructions. The internal memory 1120 may include a stored program area and a stored data area. The storage program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data (such as audio data, phone book, etc.) created during the use of the electronic device 1100 and the like.

The interface module 1130 can be used to connect an external storage device, such as an external hard disk, so as to expand the storage capacity of the electronic device 1100 . The external hard disk communicates with the processor 1110 through the interface module 1130 to realize the data storage function.

The power module 1140 is used to connect to the power grid, and supply power to the processor 1110, the internal memory 1120, and the like.

The wireless communication module 1150 can provide applications on the electronic device 1100 including wireless local area networks (WLAN) (eg wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.

Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one example embodiment or technique disclosed in accordance with this application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

The present disclosure also relates to apparatuses for performing operations in text. This apparatus may be specially constructed for the required purposes or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer readable medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read only memory (ROM), random access memory (RAM) , EPROM, EEPROM, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of medium suitable for storing electronic instructions, and each may be coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processors for increased computing power.

Additionally, the language used in this specification has been principally selected for readability and instructional purposes and may not have been selected to delineate or limit the disclosed subject matter. Accordingly, the present disclosure is intended to illustrate, but not to limit, the areas of the concepts discussed herein.

Claims (13)

1. A data processing method for an electronic device, comprising:

receiving a data query request;

determining a target data set corresponding to the query request, and determining at least one data subset of the target data set and an index information set corresponding to each data subset, wherein the data subsets comprise a plurality of data entities, and a part of the data entities in the plurality of data entities comprise first data of a character string type and second data of a non-character string type; and the index information set at least comprises first information for characterizing the correspondence of the first data and the string index in the data subset, and second information for characterizing the correspondence of the string index and the data entity in the data subset;

and determining target index information matched with the data query request and a target data entity corresponding to the target index information in the first information and the second information of a plurality of index information sets of the target data set.

2. The data processing method according to claim 1, wherein the first information for characterizing the correspondence between the first data in the data subset and the string index is dictionary tree information.

3. The data processing method according to claim 2, wherein the data query request includes third data of a string type;

the determining, from the first information and the second information of multiple index information sets of the target data set, target index information matching the data query request and a target data entity corresponding to the target index information includes:

determining state information for the at least one subset of data of the target dataset, the state information including a sequestration state and a growth state;

if the at least one data subset is in the sealed state, determining a target character string index corresponding to the third data as the target index information according to the dictionary tree information;

and determining the target data entity corresponding to the first target character string index according to the second relation.

4. The data processing method according to claim 3, wherein the determining, according to the dictionary tree information, that a target string index corresponding to the third data is the target index information includes:

looking up the third data in the dictionary tree information;

and determining the target character string index corresponding to the third data as the target index information according to the dictionary tree information.

5. The data processing method according to claim 3, wherein the determining, among the first information and the second information of the index information sets of the target data set, a target index information that matches the data query request and a target data entity corresponding to the target index information further comprises:

and determining the target data entity as a data query result corresponding to the data query request.

6. The data processing method of claim 1, wherein the data query request further includes fourth data of a non-string type.

7. The data processing method of claim 1, wherein the data query request comprises a query condition, the query condition comprising at least one of:

a Boolean expression;

characterizing prefix matching conditions of the character string prefixes;

an exact match condition of the string is characterized.

8. A data processing method for an electronic device, comprising:

receiving a data insertion request, wherein the data insertion request comprises a character string set to be inserted, and the character string set comprises a plurality of character string data;

and responding to the data insertion request, writing the plurality of character string data into corresponding data subsets of a target data set respectively, wherein each data subset comprises at least one data entity, and the at least one data entity comprises at least one character string data.

9. The data processing method of claim 8, further comprising:

acquiring the at least one character string data in the data subset;

according to the at least one character string data, constructing third information of the data subset, wherein the third information is used for representing the corresponding relation between the at least one character string data in the data subset and a character string index;

and according to each character string data in the first information, the corresponding character string index and the data entity corresponding to each character string data, fourth information of the data subset is constructed, wherein the fourth information is used for representing the corresponding relation between the character string index and the data entity in the data subset.

10. The data processing method according to claim 8, wherein said writing the plurality of character string data into the corresponding data subsets in response to the data insertion request comprises:

in response to the data insertion request, dividing the plurality of character string data into M character string subsets, and sending the character string subsets to corresponding M data nodes, wherein M is greater than or equal to 2;

and respectively writing the character string data in the character string subsets of the M data nodes into the corresponding data subsets.

11. An electronic device, comprising:

a memory for storing instructions for execution by one or more processors of the electronic device, an

A processor, being one of processors of an electronic device, for controlling execution of the data processing method of any one of claims 1 to 7 or the data processing method of any one of claims 8 to 10.

12. A computer-readable storage medium, characterized in that the storage medium has stored thereon instructions which, when executed on a computer, cause the computer to carry out the data processing method of any one of claims 1 to 7 or the data processing method of any one of claims 8 to 10.

13. A computer program product, characterized in that it comprises instructions for implementing the data processing method of any one of claims 1 to 7 or the data processing method of any one of claims 8 to 10.

Images