CN113239215B - Multimedia resource classification method, device, electronic device and storage medium - Google Patents

Abstract

The application provides a method and a device for classifying multimedia resources, electronic equipment and a storage medium, belongs to the technical field of computers, and particularly relates to the technical fields of computer vision, deep learning, knowledge graph and the like. The specific implementation scheme is as follows: the method comprises the steps of obtaining multimedia resources and multimedia expression vectors corresponding to the multimedia resources, determining knowledge expression vectors corresponding to the multimedia resources according to a plurality of entities in the multimedia resources and relations among the entities, and further determining categories of the multimedia resources according to the multimedia expression vectors and the knowledge expression vectors. Therefore, the deep understanding of the multimedia resources is enhanced by utilizing knowledge, and the classifying effect of the multimedia resources is improved.

Description

Multimedia resource classification method, device, electronic device and storage medium

Technical Field

The present application relates to the field of computer technology, specifically to technical fields such as computer vision, deep learning, knowledge graphs, and especially to classification methods, devices, electronic devices, and storage media for multimedia resources.

Background technique

In recent years, with the rapid development of software and hardware technologies, the current Internet content has gradually transitioned from the era of graphics and text to the era of multimedia (such as video). Every day, a large number of multimedia resources (such as UGC (User Generated Content) videos) are produced, distributed and consumed, which has brought huge pressure to the production system. In order to meet the needs of the current explosive growth of Internet multimedia resources, the classification technology of multimedia resources is used to automatically label the multimedia resources with their own tags according to the established tag system, thereby alleviating the review and distribution pressure of multimedia resources.

In the related art, content features are extracted from the multimedia resource content itself to classify the multimedia resources. However, for multimedia resources in some specific fields, the understanding of background knowledge is not considered when classifying the multimedia resources. For example, for videos in the fields of medicine, economy, etc., simply extracting features from the video content itself to classify the videos will result in poor classification results.

Summary of the invention

The present application provides a method, device, electronic device and storage medium for classifying multimedia resources.

According to one aspect of the present application, a method for classifying multimedia resources is provided, including: obtaining a multimedia resource to be processed, and a multimedia representation vector corresponding to the multimedia resource; determining multiple entities in the multimedia resource and the relationship between the multiple entities; determining a knowledge representation vector corresponding to the multimedia resource based on the multiple entities and the relationship between the multiple entities; determining a category of the multimedia resource based on the multimedia representation vector and the knowledge representation vector.

According to another aspect of the present application, a multimedia resource classification device is provided, including: an acquisition module, used to acquire the multimedia resource to be processed, and a multimedia representation vector corresponding to the multimedia resource; a first determination module, used to determine multiple entities in the multimedia resource and the relationship between the multiple entities; a second determination module, used to determine the knowledge representation vector corresponding to the multimedia resource based on the multiple entities and the relationship between the multiple entities; a third determination module, used to determine the category of the multimedia resource based on the multimedia representation vector and the knowledge representation vector.

According to another aspect of the present application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the classification method of multimedia resources as described above.

According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the multimedia resource classification method as described above.

According to another aspect of the present application, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the multimedia resource classification method as described above.

It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

FIG1 is a schematic diagram of a first embodiment of the present application;

FIG2 is a schematic diagram of a second embodiment of the present application;

3 is a schematic diagram of a process of determining a knowledge representation vector corresponding to a multimedia resource according to an embodiment of the present application;

FIG4 is a schematic diagram according to a third embodiment of the present application;

5 is a schematic diagram of a process of determining a knowledge representation vector corresponding to a multimedia resource according to an embodiment of the present application;

FIG6 is a schematic diagram of a fourth embodiment of the present application;

7 is a schematic diagram of determining weights of various media resources in multimedia resources according to an embodiment of the present application;

FIG8 is a schematic diagram of a fifth embodiment of the present application;

FIG9 is a schematic diagram of a sixth embodiment of the present application;

FIG10 is a schematic diagram of input of a Transformer according to an embodiment of the present application;

FIG11 is a schematic diagram of a seventh embodiment of the present application;

FIG12 is a schematic diagram of classification of multimedia resources according to an embodiment of the present application;

FIG13 is a schematic diagram of an eighth embodiment of the present application;

FIG. 14 is a block diagram of an electronic device used to implement the method for classifying multimedia resources in an embodiment of the present application.

Detailed ways

The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

In recent years, with the rapid development of software and hardware technologies, the current Internet content has slowly transitioned from the era of graphics and text to the era of multimedia (such as video). Every day, a large number of multimedia resources (such as UGC (User Generated Content) videos) are produced, distributed and consumed, which has brought huge pressure to the production system. In order to meet the needs of the current explosive growth of Internet multimedia resources, the classification technology of multimedia resources is used to automatically label the original multimedia resources with the corresponding tags according to the established tag system, thereby alleviating the review and distribution pressure of multimedia resources.

In the related art, taking multimedia resources as videos as an example, the following three methods are mainly used for classification: method (1) manual labeling: using manpower to label videos; method (2) video classification algorithm based on key frames; method (3) video classification algorithm based on multimodal fusion; however, the usage scenarios of the above three solutions are different, and the problems and shortcomings they bring are also different. The above method (1) is time-consuming, and as the amount of video increases, the labor cost required is extremely high; method (2) only uses the key frame information in the video, lacks the supplement of other modal information, and easily leads to a one-sided understanding of the video content; method (3) uses information from multiple modes of the video, but cannot form an effective content understanding of the video field that requires background knowledge.

In response to the above problems, the present application provides a method, device, electronic device and storage medium for classifying multimedia resources.

FIG1 is a schematic diagram of the first embodiment of the present application. It should be noted that the multimedia resource classification method of the embodiment of the present application can be applied to the multimedia resource classification device of the embodiment of the present application, and the device can be configured in an electronic device. The electronic device can be a mobile terminal, for example, a mobile phone, a tablet computer, a personal digital assistant, and other hardware devices with various operating systems.

As shown in FIG1 , the multimedia resource classification method may include:

Step 101: Acquire multimedia resources to be processed and multimedia representation vectors corresponding to the multimedia resources.

In the embodiment of the present application, resources including various media such as video, audio, and text may be used as multimedia resources, and the multimedia resources to be processed may be obtained by downloading from the network, uploading by users, and the like.

In order to understand the content of multimedia resources from multiple perspectives and improve the classification effect of multimedia resources, the multimedia representation vector corresponding to the multimedia resources to be processed can be obtained. For example, various media resources in the multimedia resources to be processed are represented by vectors, and the multimedia representation vector corresponding to the multimedia resources to be processed is determined according to the weight corresponding to each resource. For another example, various media resources in the multimedia resources to be processed are represented by vectors, and the weighted representation vector of the multimedia resources is determined according to the weight corresponding to each resource. Then, the multimedia resources are feature extracted to obtain the multimedia resource feature vector sequence corresponding to the multimedia resources. According to the weighted representation vector of the multimedia resources and the feature vector sequence of the multimedia resources, the multimedia representation vector corresponding to the multimedia resources is obtained.

Step 102: Determine multiple entities in the multimedia resource and the relationship between the multiple entities.

It is understandable that the multimedia resources to be processed may include multiple media resources such as video, audio, text, etc., and the video, audio and text may include different types of entities, such as entities such as people and things.

In an embodiment of the present application, multiple entities in multimedia resources can be extracted to determine the multiple entities in the multimedia resources. After determining the multiple entities in the multimedia resources, the relationship between the multiple entities can be determined based on the corresponding entity knowledge graph.

Step 103: Determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationship between the multiple entities.

Optionally, after determining multiple entities in the multimedia resources and the relationships between the multiple entities, a knowledge graph can be constructed based on the multiple entities in the multimedia resources and the relationships between the multiple entities, and each node in the knowledge graph can be encoded, so as to determine the knowledge representation vector corresponding to the multimedia resources.

Step 104: Determine the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector.

In an embodiment of the present application, the multimedia representation vector and the knowledge representation vector may be fused, and the category of the multimedia resource may be determined based on the fused result.

In summary, by obtaining multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and according to multiple entities in the multimedia resources and the relationship between the multiple entities, the knowledge representation vector corresponding to the multimedia resources is determined, and then the category of the multimedia resources is determined according to the multimedia representation vector and the knowledge representation vector. Thus, the use of knowledge enhances the deep understanding of multimedia resources and improves the classification effect of multimedia resources.

In order to more accurately determine the relationship between multiple entities in the multimedia resources, as shown in FIG2, FIG2 is a schematic diagram according to the second embodiment of the present application. In the embodiment of the present application, multiple entities in the multimedia resources can be extracted, and the relationship between the multiple entities can be determined based on the entity knowledge graph. The steps of the embodiment shown in FIG2 are as follows:

Step 201: Acquire the multimedia resources to be processed and the multimedia representation vectors corresponding to the multimedia resources.

Step 202: extract multiple entities from the multimedia resources.

In an embodiment of the present application, a multimedia resource may include multiple entities, wherein the multiple entities may be acquired by one or more of the following methods: performing face detection on each frame image of the video in the multimedia resource to extract the entity; performing optical character recognition on each frame image of the video in the multimedia resource to extract the entity; performing named entity recognition on the text in the multimedia resource to extract the entity in the text.

That is to say, in order to more accurately obtain multiple entities in multimedia resources, different methods are used to obtain entities in different media in multimedia resources. For example, face detection technology can be used to identify celebrity entities appearing in each frame image of the video in the multimedia resources, optical character recognition (OCR) technology can be used to identify subtitles appearing in each frame image of the video in the multimedia resources to obtain program names or other valid entities, and named entity recognition (NER) technology can be used to identify entity concepts contained in video titles in texts in multimedia resources.

Step 203, querying the entity knowledge graph based on the multiple entities to determine the relationship between the multiple entities.

In order to more accurately determine the relationship between multiple entities in the multimedia resources, after extracting multiple entities in the multimedia resources, an entity query can be performed on a preset entity knowledge graph to determine the relationship between the multiple entities.

Step 204: Determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationships between the multiple entities.

For example, as shown in Figure 3, taking the multimedia resource as a video, multiple entities in the video can be obtained through face detection technology, OCR technology, and NER technology. A knowledge graph can be constructed based on the multiple entities in the multimedia resources and the relationship between the multiple entities. Each node in the knowledge graph is encoded, so that the knowledge representation vector corresponding to the multimedia resource can be determined.

Step 205: Determine the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector.

In the embodiment of the present application, steps 201 and 205 can be implemented in any way in the embodiments of the present application respectively. The embodiment of the present application does not limit this and will not be repeated.

In summary, by obtaining multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and according to multiple entities in the multimedia resources and the relationship between the multiple entities, the knowledge representation vector corresponding to the multimedia resources is determined, and then the category of the multimedia resources is determined according to the multimedia representation vector and the knowledge representation vector. Thus, the use of knowledge enhances the deep understanding of multimedia resources and improves the classification effect of multimedia resources.

In order to accurately determine the knowledge representation vector, better understand the multimedia resources in depth, and improve the classification effect of multimedia resources. As shown in Figure 4, Figure 4 is a schematic diagram according to the third embodiment of the present application. In the embodiment of the present application, the representation vector of each entity can be determined according to the multiple entities in the multimedia resources and the relationship between the multiple entities, and the knowledge representation vector can be determined according to the representation vector of each entity. The steps of the embodiment shown in Figure 4 are as follows:

Step 401: Acquire the multimedia resources to be processed and the multimedia representation vectors corresponding to the multimedia resources.

Step 402: Determine multiple entities in the multimedia resource and the relationship between the multiple entities.

Step 403: Determine a representation vector of each entity according to the multiple entities and the relationships between the multiple entities.

In an embodiment of the present application, a knowledge graph can be formed based on multiple entities in a multimedia resource and the relationships between multiple entities. For example, each node in the knowledge graph is an entity in the multimedia resource, and the edge in the knowledge graph is the relationship between entities in the multimedia resource. Then, according to the knowledge graph embedding learning algorithm, each node in the knowledge graph is encoded to obtain a representation vector for each entity. Among them, the knowledge graph embedding learning algorithm may include but is not limited to TranseE, TranseH, etc.

Step 404: determine a knowledge representation vector based on the representation vector of each entity.

As an example, a global representation of a knowledge graph formed based on multiple entities and the relationships between multiple entities in multimedia resources can be obtained based on a graph neural network algorithm based on graph collapse. As shown in Figure 5, the nodes of the knowledge graph are learned through GNN (Graph Neural Networks) to obtain the probability that each node belongs to each node cluster. The knowledge graph (such as the Original graph in Figure 5) is collapsed according to this probability distribution to reduce the number of clusters. This process is repeated until the final super node is obtained. This super node is the global representation of the knowledge graph, that is, this super node is the knowledge representation vector corresponding to the multimedia resource.

Step 405: Determine the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector.

In the embodiment of the present application, steps 401-402 and 405 can be implemented in any way in the embodiments of the present application respectively. The embodiment of the present application does not limit this and will not be repeated.

In summary, by obtaining multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and according to multiple entities in the multimedia resources and the relationship between the multiple entities, the knowledge representation vector corresponding to the multimedia resources is determined, and then the category of the multimedia resources is determined according to the multimedia representation vector and the knowledge representation vector. Thus, the use of knowledge enhances the deep understanding of multimedia resources and improves the classification effect of multimedia resources.

In order to understand the content of multimedia resources from multiple perspectives and improve the classification effect of multimedia resources, as shown in Figure 6, Figure 6 is a schematic diagram according to the fourth embodiment of the present application. In the embodiment of the present application, the multimedia representation vector can be determined according to the representation vectors and weights of various media resources in the multimedia resources. The steps of the embodiment shown in Figure 6 are as follows:

Step 601, obtaining multimedia resources.

Optionally, multimedia resources may be obtained by downloading from the network, uploading by users, and the like.

Step 602: Determine the representation vectors and weights of various media resources in the multimedia resources.

It is understandable that multimedia resources may include but are not limited to resources of various media such as video, audio, and text, among which video can be decomposed into multiple continuous single-frame pictures, each frame of the picture can correspond to a vector, and then the video can correspond to a sequence vector. For example, the video can be deframed to obtain continuous pictures, each frame of the picture can correspond to a vector, and the video can correspond to a sequence vector. Audio can be decomposed into multiple continuous audios with the same time, each audio can correspond to a vector, and then the audio can correspond to a sequence vector. For example, 5 seconds of audio can be decomposed into 5 continuous audios, each audio corresponding to 1 second, each audio can correspond to a vector, and the audio can correspond to a sequence vector. Text can be decomposed into multiple characters, each character can correspond to a vector, and text can correspond to a sequence vector. In this way, the sequence vectors corresponding to various media resources in the multimedia resources can be obtained.

As an example, the sequence vectors corresponding to various media resources in the multimedia resources may be pooled to obtain representation vectors of the various media resources. For example, the sequence vectors corresponding to video, audio, and text may be pooled to obtain a video representation vector, an audio representation vector, and a text representation vector.

In the embodiment of the present application, after the representation vectors of various media resources are obtained, the weights of the various media resources may be obtained according to the representation vectors of the various media resources and a preset context vector.

For example, as shown in FIG7 , the similarities between the preset context vector and the representation vectors of various media resources can be calculated respectively according to formula (1), and the similarities can be normalized according to formula (2) to obtain the weights of various media resources.

Wherein, E _c represents a preset context vector, E _i represents a representation vector of various media resources, i represents the number of media resources in the multimedia resources, E _t may represent a representation vector of a media resource (for example, E _t may be a video representation vector, or, E _t may be an audio representation vector, or, E _t may be a text representation vector), W represents a preset value, A transformation matrix representing the representation vector of a media resource.

Step 603: Perform weighted summation processing according to the representation vectors and weights of various media resources to obtain a weighted representation vector.

In order to understand the content of multimedia resources from multiple perspectives and improve the classification effect of multimedia resources, the representation vectors and weights of various media resources may be weighted and summed to obtain a weighted representation vector.

For example, taking multimedia resources including video, audio, and text as an example, the representation vectors of various media resources in the multimedia resources can be respectively expressed as E _video (video representation vector), E _audio (audio representation vector), and E _text (text representation vector), and the weights of various media resources are respectively expressed as a _v (video weight), a _a (audio weight), and a _t (text weight), and the weighted representation vector can be expressed as: weighted representation vector = a _v E _video + a _a E _audio + a _t E _text .

Step 604: Use the weighted representation vector as a multimedia representation vector.

Furthermore, a weighted summation process is performed according to the representation vectors and weights of various media resources, and the obtained weighted representation vector is used as the multimedia representation vector.

Step 605: Determine multiple entities in the multimedia resource and the relationship between the multiple entities.

Step 606: Determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationships between the multiple entities.

Step 607: Determine the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector.

In the embodiment of the present application, steps 605-607 can be implemented in any of the embodiments of the present application respectively. The embodiment of the present application does not limit this and will not be described in detail.

In summary, by obtaining multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and according to multiple entities in the multimedia resources and the relationship between the multiple entities, the knowledge representation vector corresponding to the multimedia resources is determined, and then the category of the multimedia resources is determined according to the multimedia representation vector and the knowledge representation vector. Thus, the use of knowledge enhances the deep understanding of multimedia resources and improves the classification effect of multimedia resources.

In order to more deeply understand the content of multimedia resources from multiple perspectives and further improve the classification effect of multimedia resources, as shown in Figure 8, Figure 8 is a schematic diagram according to the fifth embodiment of the present application. In the embodiment of the present application, after determining the weighted representation vector of the multimedia resource according to the representation vectors and weights of various media resources in the multimedia resource, the feature vector sequence of the multimedia resource can be extracted, and the feature vector of the multimedia resource is processed to obtain a processed feature vector sequence, and the multimedia representation vector is obtained according to the processed feature vector. The steps of the embodiment shown in Figure 8 are as follows:

Step 801: Acquire multimedia resources, and determine a weighted representation vector of the multimedia resources according to representation vectors and weights of various media resources in the multimedia resources.

Step 802: extract a feature vector sequence of multimedia resources, and add a weighted representation vector to each feature vector in the feature vector sequence to obtain a processed feature vector sequence.

As an example, feature extraction can be performed on multimedia resources to obtain multiple feature vector subsequences of media resources, and a feature vector sequence of media resources can be obtained based on the multiple feature vector subsequences, and then the feature vector sequences of media resources can be spliced to obtain a feature vector sequence of multimedia resources. The features of multimedia resources may include but are not limited to multiple media resource information features, multiple media resource type features, multiple media resource location features, etc.

Furthermore, a weighted representation vector is added to each feature vector in the feature vector sequence of the multimedia, and the addition result is used as the processed feature vector sequence.

Step 803: semantically encode the processed feature vector sequence to obtain a multimedia representation vector.

As an example, the processed feature vector sequence may be pooled to obtain a multimedia representation vector.

Step 804: determine multiple entities in the multimedia resource and the relationship between the multiple entities.

Step 805: Determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationships between the multiple entities.

Step 806: Determine the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector.

In the embodiment of the present application, steps 801, 804-806 can be implemented in any way in the embodiments of the present application respectively. The embodiment of the present application does not limit this and will not be repeated.

In summary, by obtaining multimedia resources and the multimedia representation vectors corresponding to the multimedia resources, and according to multiple entities in the multimedia resources and the relationship between the multiple entities, the knowledge representation vector corresponding to the multimedia resources is determined, and then the category of the multimedia resources is determined according to the multimedia representation vector and the knowledge representation vector. Thus, the use of knowledge enhances the deep understanding of multimedia resources and improves the classification effect of multimedia resources.

In order to facilitate the extraction of feature vector sequences of multimedia resources, and further understand the content of multimedia resources from multiple perspectives, and further improve the classification effect of multimedia resources, based on FIG8, FIG9 is a schematic diagram according to the sixth embodiment of the present application. In the embodiment of the present application, feature extraction can be performed on multimedia resources to obtain multiple feature vector subsequences of media resources, and feature vector sequences of media resources can be obtained based on the multiple feature vector subsequences, and then the feature vector sequences of media resources are spliced to obtain feature vector sequences of multimedia resources. The embodiment shown in FIG9 includes the following steps:

Step 901: Acquire multimedia resources, and determine a weighted representation vector of the multimedia resources according to representation vectors and weights of various media resources in the multimedia resources.

Step 902: For each media resource in the multimedia resources, multiple features are extracted from the media resource to obtain multiple feature vector subsequences of the media resource.

In the embodiments of the present application, the characteristics of multimedia resources may include, but are not limited to, various media resource information characteristics, various media resource type characteristics, various media resource location characteristics, and the like.

As an example, multiple media resource information features can be obtained from various media resources. Taking video as an example, for example, a convolutional neural network can be used to obtain image features from each frame of the video in the multimedia resource; taking audio as an example, since audio can also be regarded as an image after signal conversion, a convolutional neural network can be used for feature extraction; taking text as an example, after text tagging, the corresponding features can be obtained through pre-trained language models such as BERT or ERNIE.

The corresponding feature vector subsequence can be obtained based on the image features of each frame of the video, the image features of each second of the audio, and the text features of each character of the text. In order to identify the corresponding feature vector subsequence, the feature vector subsequence corresponding to each media can be identified with different data. The marked corresponding feature vector subsequence is maximized and pooled to obtain the feature vector subsequence of each media resource information. As shown in Figure 10, Figure 10 is a schematic diagram of the input of the Transformer according to an embodiment of the present application. The feature vector subsequence corresponding to the video can be expressed as The feature vector subsequence corresponding to the audio can be expressed as/> The feature vector subsequence corresponding to the text can be expressed as The numbers of feature vector subsequences corresponding to the video, audio and text may be the same or different.

In order to distinguish the information features of multiple media resources, type feature vector subsequences of multiple media resources can be set according to various media resource types in the multimedia resources. As shown in Figure 10, the type feature vector subsequence corresponding to video can be expressed as [M _v ,...,M _v ], the type feature vector subsequence corresponding to audio can be expressed as [M _a ,...,M _a ], and the type feature vector subsequence corresponding to text can be expressed as [M _t ,...,M _t ].

In order to determine the positional relationship of feature vector subsequences of multiple media resource information, the relative position vector of each position can be provided by cosine coding. Then, the relative position vector of each position is aggregated and the aggregated vector is positionally encoded to obtain position feature vector subsequences of multiple media resources. As shown in FIG10 , the position feature vector subsequence corresponding to the video can be expressed as [P _agg ,P ₁ ,...P _t ], the position feature vector subsequence corresponding to the audio can be expressed as [P _agg ,P ₁ ,...P _k ], and the position feature vector subsequence corresponding to the text can be expressed as [P _agg ,P ₁ ,...P _j ].

Step 903: Add up the multiple feature vector subsequences of the media resource to obtain the feature vector sequence of the media resource.

Furthermore, multiple feature vector subsequences of the media resource may be added together, and the addition result may be used as the feature vector sequence of the media resource.

For example, the feature vector subsequence corresponding to the video, the type feature vector subsequence corresponding to the video, and the position feature vector subsequence corresponding to the video may be added, and the addition result may be used as the feature vector sequence of the video.

For another example, the feature vector subsequence corresponding to the audio, the type feature vector subsequence corresponding to the audio, and the position feature vector subsequence corresponding to the audio may be added, and the addition result may be used as the feature vector sequence of the audio.

For another example, the feature vector subsequence corresponding to the text, the type feature vector subsequence corresponding to the text, and the position feature vector subsequence corresponding to the text may be added together, and the addition result may be used as the feature vector sequence of the text.

Step 904: concatenate the feature vector sequences of various media resources to obtain a feature vector sequence of multimedia resources.

For example, the feature vector sequence of the video, the feature vector sequence of the audio, and the feature vector sequence of the text may be concatenated, and the addition result may be used as the feature vector sequence of the multimedia resource.

Step 905: Add a weighted representation vector to each feature vector in the feature vector sequence to obtain a processed feature vector sequence.

Step 906: semantically encode the processed feature vector sequence to obtain a multimedia representation vector.

Step 907: Determine multiple entities in the multimedia resource and the relationship between the multiple entities.

Step 908: Determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationships between the multiple entities.

Step 909: Determine the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector.

In the embodiment of the present application, steps 901, 905-909 can be implemented in any way in the embodiments of the present application respectively. The embodiment of the present application does not limit this and will not be repeated.

In summary, by obtaining multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and according to multiple entities in the multimedia resources and the relationship between the multiple entities, the knowledge representation vector corresponding to the multimedia resources is determined, and then the category of the multimedia resources is determined according to the multimedia representation vector and the knowledge representation vector. Thus, the use of knowledge enhances the deep understanding of multimedia resources and improves the classification effect of multimedia resources.

In order to automatically classify multimedia resources, meet the timeliness of multimedia resource classification, and reduce the pressure of multimedia resource review and distribution, as shown in Figure 11, Figure 11 is a schematic diagram according to the seventh embodiment of the present application. In the embodiment of the present application, knowledge is used to enhance the deep understanding of multimedia resources and determine the type of multimedia resources. The embodiment shown in Figure 11 includes the following steps:

Step 1101: Acquire the multimedia resources to be processed and the multimedia representation vectors corresponding to the multimedia resources.

Step 1102: Determine multiple entities in the multimedia resource and the relationship between the multiple entities.

Step 1103: Determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationship between the multiple entities.

Step 1104, determining the weights of the multimedia representation vector and the knowledge representation vector.

In the embodiment of the present application, the weight of the multimedia representation vector can be set to 1, and the weight of the knowledge representation vector can be obtained according to the following formula:

W _gate = sigmoid(W[K _trans ,H _kno ]+b) (3)

Among them, W _gate represents the weight of the knowledge representation vector, K _trans represents the multimedia representation vector, H _kno represents the knowledge representation vector, and b represents the preset offset.

Step 1105 , weighted processing is performed on the multimedia representation vector and the knowledge representation vector according to the weights to obtain a fused representation vector.

For example, the fusion representation vector can be obtained by the following formula:

H＝K _trans +W _gate H _kno (4)

Among them, H represents the fusion representation vector, K _trans represents the multimedia representation vector, H _kno represents the knowledge representation vector, and W _gate represents the weight of the knowledge representation vector.

Step 1106: Determine the type of multimedia resource according to the fusion representation vector.

Optionally, based on the fused representation vector, an intermediate vector for predicting each level category is determined; for each level, based on the intermediate vector of the level and the intermediate vector of the higher level, the category of the multimedia resource at the level is predicted, wherein the higher level is a level higher than the level.

It should be understood that in order to more accurately determine the type of multimedia resources, the type of multimedia resources can be represented by setting corresponding tags. The tags corresponding to multimedia resources have a hierarchical structure. For example, taking the multimedia resource as a video, the first-level tag corresponding to the video can be animation, film and television, etc. The second-level tag corresponding to the first-level tag animation can be domestic animation, foreign animation, and the second-level tag corresponding to the first-level tag film and television can be costume drama, idol drama, etc. In other words, the intermediate vector used to predict each level category can be determined based on the fusion representation vector. Among them, the intermediate vector can be the classification representation vector corresponding to the label. When predicting the category of the multimedia resource at the level, the prediction can be made based on the intermediate vector of the level and the intermediate vector above the level.

For example, as shown in Figure 12, H is the fusion representation vector. After the neural network, h _l1 and h _l2 can be obtained. h _l1 and h _l2 are the classification representation vectors of the first-level label and the second-level label. In the second-level label classification layer, the constraint relationship between the first-level and second-level labels is established by splicing the first-level label representation vector. and secondary tags/> It can be calculated according to the following formula:

Among them, w _l1 and w _l2 are preset weights, and b _l1 and b _l2 are preset offsets.

In the embodiment of the present application, steps 1101-1103 can be implemented in any of the embodiments of the present application respectively. The embodiment of the present application does not limit this and will not be repeated.

In summary, by obtaining multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and according to multiple entities in the multimedia resources and the relationship between the multiple entities, the knowledge representation vector corresponding to the multimedia resources is determined, and then the category of the multimedia resources is determined according to the multimedia representation vector and the knowledge representation vector. Thus, the use of knowledge enhances the deep understanding of multimedia resources and improves the classification effect of multimedia resources.

The multimedia resource classification method of the embodiment of the present application obtains multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and determines the knowledge representation vector corresponding to the multimedia resources according to multiple entities in the multimedia resources and the relationship between the multiple entities, and then determines the category of the multimedia resources according to the multimedia representation vector and the knowledge representation vector. Thus, the deep understanding of multimedia resources is enhanced by using knowledge, and the classification effect of multimedia resources is improved.

In order to implement the above embodiment, the embodiment of the present application also proposes a classification device for multimedia resources.

Fig. 13 is a schematic diagram of the eighth embodiment of the present application. As shown in Fig. 13 , the multimedia resource classification device 1300 includes: an acquisition module 1310 , a first determination module 1320 , a second determination module 1330 , and a third determination module 1340 .

Among them, the acquisition module 1310 is used to obtain the multimedia resources to be processed and the multimedia representation vector corresponding to the multimedia resources; the first determination module 1320 is used to determine multiple entities in the multimedia resources and the relationship between the multiple entities; the second determination module 1330 is used to determine the knowledge representation vector corresponding to the multimedia resources based on the multiple entities and the relationship between the multiple entities; the third determination module 1340 is used to determine the category of the multimedia resources based on the multimedia representation vector and the knowledge representation vector.

As a possible implementation method of the embodiment of the present application, the first determination module 1320 is specifically used to: extract the multiple entities in the multimedia resources; query the entity knowledge graph according to the multiple entities, and determine the relationship between the multiple entities.

As a possible implementation method of an embodiment of the present application, the first determination module 1320 is also used to: perform face detection on each frame image of the video in the multimedia resource to extract entities; and/or, perform optical character recognition on each frame image of the video in the multimedia resource to extract entities; and/or, perform named entity recognition on the text in the multimedia resource to extract entities in the text.

As a possible implementation method of the embodiment of the present application, the second determination module 1330 is specifically used to: determine the representation vector of each entity according to multiple entities and the relationship between the multiple entities; determine the knowledge representation vector according to the representation vector of each entity.

As a possible implementation method of an embodiment of the present application, the acquisition module 1310 is specifically used to: acquire multimedia resources; determine the representation vectors and weights of various media resources in the multimedia resources; perform weighted summation processing based on the representation vectors and weights of various media resources to obtain a weighted representation vector; and use the weighted representation vector as a multimedia representation vector.

As a possible implementation method of an embodiment of the present application, the acquisition module 1310 is also used to: acquire multimedia resources, and determine a weighted representation vector of the multimedia resources based on the representation vectors and weights of various media resources in the multimedia resources; extract a feature vector sequence of the multimedia resources, and add the weighted representation vector to each feature vector in the feature vector sequence to obtain a processed feature vector sequence; and perform semantic encoding on the processed feature vector sequence to obtain a multimedia representation vector.

As a possible implementation method of an embodiment of the present application, the acquisition module 1310 is also used to: for each media resource in the multimedia resources, perform multiple feature extraction on the media resources to obtain multiple feature vector subsequences of the media resources; add the multiple feature vector subsequences of the media resources to obtain a feature vector sequence of the media resources; splice the feature vector sequences of various media resources to obtain a feature vector sequence of the multimedia resources.

As a possible implementation method of an embodiment of the present application, the third determination module 1340 is specifically used to: determine the weights of the multimedia representation vector and the knowledge representation vector; perform weighted processing on the multimedia representation vector and the knowledge representation vector according to the weights to obtain a fused representation vector; and determine the type of multimedia resource according to the fused representation vector.

As a possible implementation method of an embodiment of the present application, the third determination module 1340 is also used to: determine the intermediate vector used to predict each level category based on the fused representation vector; for each level, predict the level category of the multimedia resource based on the intermediate vector of the level and the intermediate vector of the high level, wherein the high level is a level higher than the level.

The multimedia resource classification device of the embodiment of the present application obtains multimedia resources and multimedia representation vectors corresponding to the multimedia resources, and determines the knowledge representation vector corresponding to the multimedia resources according to multiple entities in the multimedia resources and the relationship between the multiple entities, and then determines the category of the multimedia resources according to the multimedia representation vector and the knowledge representation vector. Thus, the deep understanding of multimedia resources is enhanced by using knowledge, and the classification effect of multimedia resources is improved.

According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

Figure 14 shows a schematic block diagram of an example electronic device 1400 that can be used to implement an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and/or required herein.

As shown in FIG. 14 , the device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input/output (I/O) interface 1405 is also connected to the bus 1404.

A number of components in the device 1400 are connected to the I/O interface 1405, including: an input unit 1406, such as a keyboard, a mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a disk, an optical disk, etc.; and a communication unit 1409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the device 1400 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The computing unit 1401 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above, such as a classification method for multimedia resources. For example, in some embodiments, the classification method for multimedia resources may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and/or installed on the device 1400 via the ROM 1402 and/or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, one or more steps of the classification method for multimedia resources described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to execute the multimedia resource classification method in any other appropriate manner (for example, by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions/operations specified in the flow chart and/or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other. The server may also be a server of a distributed system, or a server combined with a blockchain.

It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning/deep learning, big data processing technology, knowledge graph technology, and other major directions.

In addition, the acquisition, storage and application of information involved in the technical solution of this application are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution proposed in this application can be achieved, and this document does not limit this.

The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims (21)

1. A method for classifying multimedia resources, comprising: Acquire multimedia resources to be processed and multimedia representation vectors corresponding to the multimedia resources, wherein various media resources in the multimedia resources to be processed are represented by vectors, and the multimedia representation vectors corresponding to the multimedia resources to be processed are determined according to the weights corresponding to each resource; Determining a plurality of entities in the multimedia resource and relationships between the plurality of entities; Determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationships between the multiple entities, wherein a knowledge graph is formed according to the multiple entities in the multimedia resource and the relationships between the multiple entities, and each node in the knowledge graph is encoded to determine the knowledge representation vector corresponding to the multimedia resource; The category of the multimedia resource is determined according to the multimedia representation vector and the knowledge representation vector, the multimedia representation vector and the knowledge representation vector are fused, and the category of the multimedia resource is determined according to the fused result. 2. The method according to claim 1, wherein the determining a plurality of entities in the multimedia resource and the relationship between the plurality of entities comprises: extracting the plurality of entities from the multimedia resource; An entity knowledge graph is queried according to the multiple entities to determine the relationships between the multiple entities. 3. The method according to claim 2, wherein the method of extracting the multiple entities in the multimedia resource comprises at least one of the following methods: Performing face detection on each frame image of the video in the multimedia resource to extract the entity; Performing optical character recognition on each frame image of the video in the multimedia resource to extract the entity; Perform named entity recognition on the text in the multimedia resource and extract entities in the text. 4. The method according to claim 1, wherein determining the knowledge representation vector corresponding to the multimedia resource according to the plurality of entities and the relationship between the plurality of entities comprises: Determine a representation vector of each entity according to the multiple entities and the relationship between the multiple entities; The knowledge representation vector is determined according to the representation vector of each entity. 5. The method according to claim 1, wherein the step of obtaining the multimedia resource to be processed and the multimedia representation vector corresponding to the multimedia resource comprises: Acquiring the multimedia resource; Determining representation vectors and weights of various media resources in the multimedia resources; Performing weighted summation processing according to the representation vectors and weights of the various media resources to obtain a weighted representation vector; The weighted representation vector is used as the multimedia representation vector. 6. The method according to claim 1, wherein the step of obtaining the multimedia resource to be processed and the multimedia representation vector corresponding to the multimedia resource comprises: Acquire the multimedia resource, and determine a weighted representation vector of the multimedia resource according to representation vectors and weights of various media resources in the multimedia resource; Extracting a feature vector sequence of the multimedia resource, and adding the weighted representation vector to each feature vector in the feature vector sequence to obtain a processed feature vector sequence; The processed feature vector sequence is semantically encoded to obtain the multimedia representation vector. 7. The method according to claim 6, wherein the step of extracting the feature vector sequence of the multimedia resource comprises: For each media resource in the multimedia resources, extract multiple features from the media resource to obtain multiple feature vector subsequences of the media resource; Adding and processing the multiple feature vector subsequences of the media resource to obtain the feature vector sequence of the media resource; The feature vector sequences of various media resources are concatenated to obtain the feature vector sequence of the multimedia resource. 8. The method according to claim 1, wherein determining the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector comprises: Determining weights of the multimedia representation vector and the knowledge representation vector; According to the weight, weighted processing is performed on the multimedia representation vector and the knowledge representation vector to obtain a fusion representation vector; The type of the multimedia resource is determined according to the fused representation vector. 9. The method according to claim 8, wherein determining the category of the multimedia resource according to the fused representation vector comprises: Determining, based on the fused representation vector, an intermediate vector for predicting each level category; For each level, the category of the multimedia resource at the level is predicted according to the intermediate vector of the level and the intermediate vector of a higher level, wherein the higher level is a level higher than the level. 10. A multimedia resource classification device, comprising: An acquisition module, used to acquire multimedia resources to be processed and multimedia representation vectors corresponding to the multimedia resources, wherein various media resources in the multimedia resources to be processed are represented by vectors, and the multimedia representation vectors corresponding to the multimedia resources to be processed are determined according to the weights corresponding to each resource; A first determining module, configured to determine a plurality of entities in the multimedia resource and the relationship between the plurality of entities; A second determination module is used to determine a knowledge representation vector corresponding to the multimedia resource according to the multiple entities and the relationships between the multiple entities, wherein a knowledge graph is formed according to the multiple entities in the multimedia resource and the relationships between the multiple entities, and each node in the knowledge graph is encoded to determine the knowledge representation vector corresponding to the multimedia resource; The third determination module is used to determine the category of the multimedia resource according to the multimedia representation vector and the knowledge representation vector, fuse the multimedia representation vector and the knowledge representation vector, and determine the category of the multimedia resource according to the fused result. 11. The device according to claim 10, wherein the first determining module is specifically configured to: extracting the plurality of entities from the multimedia resource; An entity knowledge graph is queried according to the multiple entities to determine the relationships between the multiple entities. 12. The apparatus according to claim 11, wherein the first determining module is further configured to: Performing face detection on each frame image of the video in the multimedia resource to extract the entity; and / or, Performing optical character recognition on each frame image of the video in the multimedia resource to extract the entity; and / or, Perform named entity recognition on the text in the multimedia resource and extract entities in the text. 13. The device according to claim 10, wherein the second determining module is specifically configured to: Determine a representation vector of each entity according to the multiple entities and the relationship between the multiple entities; The knowledge representation vector is determined according to the representation vector of each entity. 14. The device according to claim 10, wherein the acquisition module is specifically used to: Acquiring the multimedia resource; Determining representation vectors and weights of various media resources in the multimedia resources; Performing weighted summation processing according to the representation vectors and weights of the various media resources to obtain a weighted representation vector; The weighted representation vector is used as the multimedia representation vector. 15. The device according to claim 10, wherein the acquisition module is further used for: Acquire the multimedia resource, and determine a weighted representation vector of the multimedia resource according to representation vectors and weights of various media resources in the multimedia resource; Extracting a feature vector sequence of the multimedia resource, and adding the weighted representation vector to each feature vector in the feature vector sequence to obtain a processed feature vector sequence; The processed feature vector sequence is semantically encoded to obtain the multimedia representation vector. 16. The device according to claim 15, wherein the acquisition module is further used for: For each media resource in the multimedia resources, extract multiple features from the media resource to obtain multiple feature vector subsequences of the media resource; Adding and processing the multiple feature vector subsequences of the media resource to obtain the feature vector sequence of the media resource; The feature vector sequences of various media resources are concatenated to obtain the feature vector sequence of the multimedia resource. 17. The device according to claim 10, wherein the third determining module is specifically configured to: Determining weights of the multimedia representation vector and the knowledge representation vector; According to the weight, weighted processing is performed on the multimedia representation vector and the knowledge representation vector to obtain a fusion representation vector; The type of the multimedia resource is determined according to the fused representation vector. 18. The apparatus according to claim 17, wherein the third determining module is further configured to: Determining, based on the fused representation vector, an intermediate vector for predicting each level category; For each level, the category of the multimedia resource at the level is predicted according to the intermediate vector of the level and the intermediate vector of a higher level, wherein the higher level is a level higher than the level. 19. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9. 20. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9. 21. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.