CN112015923A - Multi-mode data retrieval method, system, terminal and storage medium - Google Patents

Abstract

The invention discloses a multi-mode data retrieval method, a system, a terminal and a storage medium. The method comprises the following steps: acquiring historical multi-modal data, wherein the historical multi-modal data at least comprises picture data and text data; training a cross-modal retrieval model according to the historical multi-modal data; the cross-modal retrieval model at least comprises a picture modal retrieval model and a text modal retrieval model; inputting the data to be retrieved into the cross-modal retrieval model, wherein the cross-modal retrieval model retrieves the data to be retrieved through the picture modal retrieval model and the text modal retrieval model respectively to obtain a candidate set of similar data files of the data to be retrieved, and performs similarity sorting on the candidate set of similar data files to obtain a data file with the highest similarity with the data to be retrieved. The invention can well search multi-mode data such as pictures, texts and the like, and improves the search accuracy and the search efficiency.

Description

A kind of multimodal data retrieval method, system, terminal and storage medium

technical field

The present invention relates to the technical field of data retrieval, in particular to a multimodal data retrieval method, system, terminal and storage medium.

Background technique

With the rapid development of network technology, multimodal documents containing data such as text and images appear in people's daily life on a large scale. In the world of information, these data resources of different modalities can invisibly enhance the ability of the sense organs to receive knowledge.

Due to the diversity, complexity and randomness of multimodal data, how to quickly and accurately retrieve information useful to users from a large number of multimodal documents is of great significance. However, traditional data retrieval methods usually search through keywords, which need to manually extract the corresponding keywords in advance, and because the keywords are coarse-grained, the retrieval accuracy and efficiency are relatively poor.

SUMMARY OF THE INVENTION

The present invention provides a multimodal data retrieval method, system, terminal and storage medium, which can solve the deficiencies in the prior art to a certain extent.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

A multimodal data retrieval method, comprising:

Obtaining historical multimodal data, the historical multimodal data at least includes picture data and text data;

Train a cross-modal retrieval model according to the historical multimodal data; the cross-modal retrieval model includes at least a picture modal retrieval model and a text modal retrieval model;

Input the data to be retrieved into the cross-modal retrieval model, and the cross-modal retrieval model retrieves the data to be retrieved through the image modal retrieval model and the text modal retrieval model, respectively, to obtain the data to be retrieved The similar data file candidate set is obtained, and the similarity degree is sorted on the similar data file candidate set to obtain the data file with the highest similarity with the to-be-retrieved data.

The technical solution adopted in the embodiment of the present invention further includes: the obtaining historical multimodal data further includes:

A multimodal file database is constructed, and the multimodal file database includes picture data and text data of each data file.

The technical solution adopted in the embodiment of the present invention further includes: before the training of the cross-modal retrieval model according to the historical multimodal data further includes:

Labeling the categories to which the data files in the multimodal file database belong to generate data samples for training the model.

The technical solution adopted in the embodiment of the present invention further includes: the training of a cross-modal retrieval model according to the historical multimodal data includes:

The model training includes a retrieval recall stage and a precise sorting stage, wherein:

In the retrieval and recall stage, a matching algorithm is used to perform rough screening on all data samples to obtain at least two similar data file sets of the files to be retrieved in different modalities, and then take the union of the at least two similar data file sets A candidate set of similar data files as the data to be retrieved;

In the precise sorting stage, similarity sorting is performed on the similar data file candidate set to obtain a data file with the highest similarity with the data to be retrieved.

The technical solution adopted in the embodiment of the present invention further includes: the cross-modal retrieval model respectively retrieving the data to be retrieved through the picture modal retrieval model and the text modal retrieval model includes:

Determine whether the picture data of the data to be retrieved is empty, if not, input the picture data into the picture modal retrieval model, obtain the similar data file retrieval results in the picture modal, and compare the retrieval results to the retrieval results. After sorting, get the first M retrieval results as the similar data file set S _I under the picture mode;

Judging whether the text data of the data to be retrieved is empty, if not, inputting the text data into the text modal retrieval model, obtaining the retrieval results of similar data files in the text modal, and analyzing the retrieval results After sorting, take the first M retrieval results as the similar data file set _ST under the text mode;

Take the union of sets S _I and S _T as a candidate set of similar data files of the data to be retrieved;

Sorting the similarity of the candidate set of similar data files to obtain a data file with the highest similarity with the data to be retrieved.

The technical solutions adopted in the embodiments of the present invention further include: the picture modal retrieval model is coded by using ResNet, and the text modal retrieval model is coded by using BERT.

The technical solution adopted in the embodiment of the present invention further includes: the retrieval algorithm of the text modal retrieval model includes BM25 or TFIDF algorithm, the retrieval algorithm of the picture modal retrieval model includes similarity matching using the visual features of the pictures, the Visual features include color distribution, geometry or texture.

Another technical solution adopted by the embodiment of the present invention is: a multimodal data retrieval system, comprising:

Data collection module: used to obtain historical multimodal data, the historical multimodal data includes at least picture data and text data;

Model building module: used to train a cross-modal retrieval model according to the historical multimodal data; the cross-modal retrieval model includes at least a picture modal retrieval model and a text modal retrieval model;

Data retrieval module: used to input the data to be retrieved into the cross-modal retrieval model, and the cross-modal retrieval model retrieves the data to be retrieved through the image modal retrieval model and the text modal retrieval model, respectively, Obtaining a candidate set of similar data files of the data to be retrieved, and sorting the candidate sets of similar data files by similarity to obtain a data file with the highest similarity with the data to be retrieved.

Another technical solution adopted by the embodiments of the present invention is: a terminal, the terminal includes a processor and a memory coupled to the processor, wherein,

The memory stores program instructions for implementing the above-mentioned multimodal data retrieval method;

The processor is configured to execute the program instructions stored in the memory to perform the multimodal data retrieval operation.

Another technical solution adopted by the embodiments of the present invention is: a storage medium storing program instructions executable by a processor, where the program instructions are used to execute the above-mentioned multimodal data retrieval method.

The beneficial effects of the present invention are: the multi-modal data retrieval method, system, terminal and storage medium of the embodiments of the present invention construct a cross-modal retrieval model based on data files of different modalities, and directly input data files of different modalities, Then, the retrieval results of the corresponding modalities are output, so as to realize an end-to-end retrieval scheme, which can well retrieve multi-modal data such as pictures and texts, and improve retrieval accuracy and retrieval efficiency.

Description of drawings

1 is a schematic flowchart of a multimodal data retrieval method according to a first embodiment of the present invention;

2 is a schematic flowchart of a multimodal data retrieval method according to a second embodiment of the present invention;

3 is a schematic structural diagram of a multimodal data retrieval system according to an embodiment of the present invention;

4 is a schematic structural diagram of a terminal according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a storage medium according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms "first", "second" and "third" in the present invention are only used for description purposes, and should not be understood as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature defined as "first", "second", "third" may expressly or implicitly include at least one of that feature. In the description of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship between various components under a certain posture (as shown in the accompanying drawings). , motion situation, etc., if the specific posture changes, the directional indication also changes accordingly. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

Multimodal data usually includes data in different forms such as text, image, voice, video, etc. For the same type of data, although data of different modalities will show heterogeneity of underlying features, their high-level semantics are related to a certain extent. For example, for a certain disease, the drug use and the medical imaging examination are similar to a large extent, that is, the text data of the drug use and the medical image data of the medical imaging examination are semantically related. Based on this feature, the embodiment of the present invention uses historical multi-modal data to train a cross-modal retrieval model, retrieves data sets of different modalities similar to the data to be retrieved through the cross-modal retrieval model, and then retrieves data from different modalities The intersection of the sets serves as the final retrieval result.

For convenience of description, the following embodiments of the present invention only take the two most commonly used modal data, picture and text, as examples for specific description. It can be understood that the present invention is also applicable to the retrieval of other modal data such as voice and video.

Specifically, please refer to FIG. 1 , which is a schematic flowchart of the multimodal data retrieval method according to the first embodiment of the present invention. The multimodal data retrieval method according to the first embodiment of the present invention includes the following steps:

S10: Obtain historical multimodal data, and build a multimodal file database based on the historical multimodal data;

In this step, the multimodal file database includes multimodal data such as pictures and texts of each data file. Assuming that the number of data files collected in the multimodal file database is N, the data sets contained in the database are {(I ₁ ,T ₁ ),(I ₂ ,T ₂ ),(I ₃ ,T ₃ ),… ,( _{IN ,T N )} }, where (I _i ,T _i ) represents the image-text pair of the ith data file.

S11: Label the categories to which a certain number of data files in the multimodal file database belong, and generate data samples for training the model;

In this step, taking a data file of medical data type as an example, the labels of the categories in the data file include disease name, medication type, imaging examination type, etc. By manually labeling the category of each data file, the data belonging to the same category can be marked Documents are similar.

S12: Train a cross-modal retrieval model according to the data samples, and retrieve the image data and text data of the file to be retrieved through the cross-modal retrieval model, to obtain a data file with the highest similarity to the file to be retrieved;

In this step, the cross-modal retrieval model includes at least a text modal retrieval model and a picture modal retrieval model. In this embodiment of the present invention, the Pairwise mode is used for model training, and the training process includes two stages: retrieval recall and precise sorting:

In the retrieval and recall stage, the matching algorithm is used to roughly screen all data samples to obtain a relatively small candidate set of similar data files; the retrieval and recall stage is a single-modal retrieval, that is, the image data of the files to be retrieved are retrieved from Similar image data files are retrieved from the image modal retrieval model, and similar text data files are retrieved from the text modal retrieval model based on the text data of the files to be retrieved, and the files to be retrieved in the image modal and text modalities are obtained respectively. Then, the union of the two similar data file sets is taken as the similar data file candidate set of the files to be retrieved. Assuming that the size of the candidate set of similar data files filtered through the retrieval and recall stage is K, the set of data files corresponding to the candidate set is {(I ₁ ,T ₁ ),(I ₂ ,T ₂ ),…,(I _K , T _K )}.

In the precise sorting stage, the similarity sorting of similar data file candidate sets obtained in the retrieval and recall stage is carried out, and the data files with the highest similarity to the files to be retrieved are obtained; among them, the precise sorting stage is designed based on the idea of Learning to Rank (sort learning). , the optimization goal of the precise sorting stage is the degree of matching between text data and image data. In Pairwise mode, Hinge Loss is used as the loss function:

Based on the above training modes, the image modal retrieval model and the text modal retrieval model are obtained respectively. The image modal retrieval model adopts the deep learning pre-trained image model ResNet for encoding, and the text modal retrieval model adopts BERT (Bidirectional Encoder Representations from Transformers, deep learning). pre-trained language model) for encoding, as shown below, the encoded embedding vectors of image I _i and text T _i are I _Ei and T _Ei respectively:

T _Ei =BERT(T _i )

I _Ei =ResNet(I _i )

In the embodiment of the present invention, the retrieval algorithm of the text modal retrieval model includes but is not limited to the BM25 or TFIDF algorithm, and the retrieval algorithm of the image modal retrieval model includes using simple visual features such as color distribution, geometric shape, texture and other simple visual features of pictures to perform similarity matching. .

Please refer to FIG. 2 , which is a schematic flowchart of a multimodal data retrieval method according to a second embodiment of the present invention. The multimodal data retrieval method of the second embodiment of the present invention includes the following steps:

S20: Select the file to be retrieved, and obtain the picture and text data of the file to be retrieved;

S21: Input the image and text data into the trained cross-modal retrieval model;

S22: Retrieve the image data and the text data separately through the cross-modal retrieval model to obtain a candidate set of similar data files of the files to be retrieved, and sort the candidate sets of similar data files by similarity, and obtain the file with the highest similarity to the file to be retrieved. data files;

In this step, the retrieval method of the cross-modal retrieval model specifically includes:

1. Determine whether the image data of the file to be retrieved is empty. If it is not empty, input the image data into the image modal retrieval model to obtain the retrieval results of the similar data files of the image modal, and retrieve the similar data files of the image modal. After sorting, get the first M retrieval results as the similar data file set S _I under the picture mode;

2. Determine whether the text data of the file to be retrieved is empty, if not, input the text data into the text modal retrieval model to obtain the retrieval results of similar data files in the text modal, and sort the retrieval results, take the top The M retrieval results are used as a set of similar data files _ST in the text mode;

3. Take the union of sets S _I and S _T as a candidate set of similar data files of the files to be retrieved;

4. Sort the similarity of the candidate set of similar data files, and obtain the retrieval result of the data file with the highest similarity with the file to be retrieved.

In the above, the value of M can be set according to the actual operation.

To sum up, the multi-modal data retrieval method of the embodiment of the present invention constructs a cross-modal retrieval model based on data files of different modalities, directly inputting data files of different modalities, and then outputting the retrieval results of the corresponding modalities, thereby The end-to-end retrieval scheme is implemented, which can handle multi-modal data such as pictures and texts well, and improves retrieval accuracy and retrieval efficiency.

In an optional implementation manner, the results of the multimodal data retrieval method may also be uploaded to the blockchain.

Specifically, the corresponding summary information is obtained based on the result of the multimodal data retrieval method. Specifically, the summary information is obtained by hashing the result of the multimodal data retrieval method, for example, using the sha256s algorithm. get. Uploading summary information to the blockchain ensures its security and fairness and transparency to users. The user can download the summary information from the blockchain in order to verify whether the results of the multimodal data retrieval method described have been tampered with. The blockchain referred to in this example is a new application mode of computer technology such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Please refer to FIG. 3 , which is a schematic structural diagram of a multimodal data retrieval system according to an embodiment of the present invention. The multimodal data retrieval system 40 according to the embodiment of the present invention includes:

Data acquisition module 41: used to acquire historical multimodal data, and build a multimodal file database based on the historical multimodal data; wherein, the multimodal file database includes pictures, texts and other multimodal data of each data file. Assuming that the number of data files collected in the multimodal file database is N, the data sets contained in the database are {(I ₁ ,T ₁ ),(I ₂ ,T ₂ ),(I ₃ ,T ₃ ),… ,( _{IN ,T N )} }, where (I _i ,T _i ) represents the image-text pair of the ith data file.

Model building module 42: used to train a cross-modal retrieval model according to the data samples in the multi-modal file database; wherein, the model training method is specifically: first, mark the category to which a certain number of data files in the multi-modal file database belong , generate data samples for training the model; then, train a cross-modal retrieval model based on the data samples;

In the embodiment of the present invention, the cross-modal retrieval model includes a text modal retrieval model and a picture modal retrieval model. The embodiment of the present invention adopts the Pairwise mode for model training, and the training process includes two stages: retrieval recall and precise sorting:

In the retrieval and recall stage, the matching algorithm is used to roughly screen all data samples to obtain a relatively small candidate set of similar data files; the retrieval and recall stage is a single-modal retrieval, that is, the image data of the files to be retrieved are retrieved from Similar image data files are retrieved from the image modal retrieval model, and similar text data files are retrieved from the text modal retrieval model based on the text data of the files to be retrieved, and the files to be retrieved in the image modal and text modalities are obtained respectively. Then, the union of the two similar data file sets is taken as the similar data file candidate set of the files to be retrieved. Assuming that the size of the candidate set of similar data files obtained through the retrieval and recall stage is K, the data file set corresponding to the candidate set is {(I ₁ ,T ₁ ),(I ₂ ,T ₂ ),…,(I _K , T _K )}.

In the precise sorting stage, the similarity sorting of similar data file candidate sets obtained in the retrieval and recall stage is carried out, and the data files with the highest similarity to the files to be retrieved are obtained; among them, the precise sorting stage is designed based on the idea of Learning to Rank (sort learning). , the optimization goal of the precise sorting stage is the degree of matching between text data and image data. In Pairwise mode, Hinge Loss is used as the loss function:

Based on the above training modes, the image modal retrieval model and the text modal retrieval model are obtained respectively. The image modal retrieval model adopts the deep learning pre-trained image model ResNet for encoding, and the text modal retrieval model adopts BERT (Bidirectional Encoder Representations from Transformers, deep learning). pre-trained language model) for encoding, as shown below, the encoded embedding vectors of image I _i and text T _i are I _Ei and T _Ei respectively:

T _Ei =BERT(T _i )

I _Ei =ResNet(I _i )

Data retrieval module 43: used to retrieve image data and text data respectively through a cross-modal retrieval model, obtain a candidate set of similar data files of the files to be retrieved, and sort the candidate sets of similar data files by similarity to obtain a candidate set of similar data files to be retrieved. The data file with the highest file similarity;

Among them, the retrieval method of the cross-modal retrieval model specifically includes:

1. Determine whether the image data of the file to be retrieved is empty. If it is not empty, input the image data into the image modal retrieval model to obtain the retrieval results of the similar data files of the image modal, and retrieve the similar data files of the image modal. After sorting, get the first M retrieval results as the similar data file set S _I under the picture mode;

2. Determine whether the text data of the file to be retrieved is empty, if not, input the text data into the text modal retrieval model to obtain the retrieval results of similar data files in the text modal, and sort the retrieval results, take the top The M retrieval results are used as a set of similar data files _ST in the text mode;

3. Take the union of sets S _I and S _T as a candidate set of similar data files of the files to be retrieved;

4. Sort the similarity of the candidate set of similar data files, and obtain the retrieval result of the data file with the highest similarity with the file to be retrieved.

Please refer to FIG. 4 , which is a schematic structural diagram of a terminal according to an embodiment of the present invention. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51 .

The memory 52 stores program instructions for implementing the multimodal data retrieval method described above.

The processor 51 is configured to execute program instructions stored in the memory 52 to perform multimodal data retrieval operations.

The processor 51 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 51 may be an integrated circuit chip with signal processing capability. The processor 51 may also be a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component . A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Please refer to FIG. 5 , which is a schematic structural diagram of a storage medium according to an embodiment of the present invention. The storage medium of the embodiment of the present invention stores a program file 61 capable of implementing all the above methods, wherein the program file 61 may be stored in the above-mentioned storage medium in the form of a software product, and includes several instructions to make a computer device (which may It is a personal computer, a server, or a network device, etc.) or a processor that executes all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes , or terminal devices such as computers, servers, mobile phones, and tablets.

In the several embodiments provided by the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the system embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units. The above is only an embodiment of the present invention, and is not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, All are similarly included in the scope of patent protection of the present invention.

Claims (10)

1. a multimodal data retrieval method, is characterized in that, comprises: Obtaining historical multimodal data, the historical multimodal data at least includes picture data and text data; Train a cross-modal retrieval model according to the historical multimodal data; the cross-modal retrieval model includes at least a picture modal retrieval model and a text modal retrieval model; Input the data to be retrieved into the cross-modal retrieval model, and the cross-modal retrieval model retrieves the data to be retrieved through the image modal retrieval model and the text modal retrieval model, respectively, to obtain the data to be retrieved The similar data file candidate set is obtained, and the similarity degree is sorted on the similar data file candidate set to obtain the data file with the highest similarity with the to-be-retrieved data. 2. The multimodal data retrieval method according to claim 1, wherein the acquiring historical multimodal data further comprises: A multimodal file database is constructed, and the multimodal file database includes picture data and text data of each data file. 3. The multimodal data retrieval method according to claim 2, wherein before the cross-modal retrieval model is trained according to the historical multimodal data, the method further comprises: Labeling the categories to which the data files in the multimodal file database belong to generate data samples for training the model. 4. The multimodal data retrieval method according to claim 3, wherein the training of a cross-modal retrieval model according to the historical multimodal data comprises: The model training includes a retrieval recall stage and a precise sorting stage, wherein: In the retrieval and recall stage, a matching algorithm is used to perform rough screening on all data samples to obtain at least two similar data file sets of the files to be retrieved in different modalities, and then take the union of the at least two similar data file sets A candidate set of similar data files as the data to be retrieved; In the precise sorting stage, similarity sorting is performed on the similar data file candidate set to obtain a data file with the highest similarity with the data to be retrieved. 5 . The multimodal data retrieval method according to claim 4 , wherein the cross-modal retrieval model retrieves the data to be retrieved respectively through the picture modal retrieval model and the text modal retrieval model. 6 . include: Determine whether the picture data of the data to be retrieved is empty, if not, input the picture data into the picture modal retrieval model, obtain the similar data file retrieval results in the picture modal, and compare the retrieval results to the retrieval results. After sorting, get the first M retrieval results as the similar data file set S _I under the picture mode; Judging whether the text data of the data to be retrieved is empty, if not, inputting the text data into the text modal retrieval model, obtaining the retrieval results of similar data files in the text modal, and analyzing the retrieval results After sorting, take the first M retrieval results as the similar data file set _ST under the text mode; Take the union of sets S _I and S _T as a candidate set of similar data files of the data to be retrieved; Sorting the similarity of the candidate set of similar data files to obtain a data file with the highest similarity with the data to be retrieved. 6 . The multi-modal data retrieval method according to claim 1 , wherein the image modal retrieval model adopts ResNet for encoding, and the text modal retrieval model adopts BERT for encoding. 7 . 7. multimodal data retrieval method according to claim 1, is characterized in that, the retrieval algorithm of described text modal retrieval model comprises BM25 or TFIDF algorithm, and the retrieval algorithm of described picture modal retrieval model comprises the retrieval algorithm that utilizes picture. Similarity matching is performed on visual features including color distribution, geometry or texture. 8. A multimodal data retrieval system, comprising: Data collection module: used to obtain historical multimodal data, the historical multimodal data includes at least picture data and text data; Model building module: used to train a cross-modal retrieval model according to the historical multimodal data; the cross-modal retrieval model includes at least a picture modal retrieval model and a text modal retrieval model; Data retrieval module: used to input the data to be retrieved into the cross-modal retrieval model, and the cross-modal retrieval model retrieves the data to be retrieved through the image modal retrieval model and the text modal retrieval model, respectively, Obtaining a candidate set of similar data files of the data to be retrieved, and performing similarity sorting on the candidate set of similar data files to obtain a data file with the highest similarity with the data to be retrieved. 9. A terminal, characterized in that the terminal comprises a processor and a memory coupled to the processor, wherein, The memory stores program instructions for implementing the multimodal data retrieval method according to any one of claims 1 to 7; The processor is configured to execute the program instructions stored in the memory to perform the multimodal data retrieval method. 10 . A storage medium, characterized in that it stores program instructions executable by a processor, and the program instructions are used to execute the multimodal data retrieval method according to any one of claims 1 to 7 .

Images