CN113992972B - Subtitle display method, device, electronic device and readable storage medium - Google Patents

Abstract

The present application discloses a subtitle display method, device, electronic device and readable storage medium, which belongs to the field of terminal technology. The subtitle display method includes: during the playback of a target video, when it is detected that a first image frame of the target video includes multiple first objects, according to an audio recognition model, a target object corresponding to first audio data is determined from multiple first objects, wherein the first audio data corresponds to the first image frame; according to the display position of the target object, subtitle information corresponding to the first audio data is displayed; wherein the audio recognition model is trained based on a second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

Description

Subtitle display method, device, electronic device and readable storage medium

Technical Field

The present application belongs to the field of terminal technology, and specifically relates to a subtitle display method, device, electronic device and readable storage medium.

Background Art

At present, subtitles have been widely used in various types of multimedia data such as movies, TV series, games, etc. By displaying subtitles, on the one hand, it is convenient for users to appreciate the sound art of the original video, and on the other hand, it is also convenient for hearing-impaired users to watch the video and understand the content of the video.

With the development of technology, it is now possible to convert audio data into subtitles in real time during video playback through recognition models. However, due to the difficulty in obtaining effective training data, the subtitles obtained in real time are usually difficult to correspond to the characters in the video, affecting the user's viewing experience.

Summary of the invention

The purpose of the embodiments of the present application is to provide a subtitle display method, device, electronic device and readable storage medium, which can solve the problem that subtitles are difficult to correspond to characters included in the video, affecting the user's experience of watching the video.

In a first aspect, an embodiment of the present application provides a subtitle display method, the method comprising:

During the playback of the target video, when it is detected that a first image frame of the target video includes a plurality of first objects, determining a target object corresponding to first audio data from the plurality of first objects according to an audio recognition model, wherein the first audio data corresponds to the first image frame;

Displaying subtitle information corresponding to the first audio data according to the display position of the target object;

The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

In a second aspect, an embodiment of the present application provides a subtitle display device, the device comprising:

a processing module, configured to determine, during the playback of a target video, a target object corresponding to first audio data from among the multiple first objects when it is detected that a first image frame of the target video includes multiple first objects according to an audio recognition model, wherein the first audio data corresponds to the first image frame;

A display module, used for displaying subtitle information corresponding to the first audio data according to a display position of the target object;

The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

In an embodiment of the present application, during the playback of a target video, for a first image frame including multiple first objects, by using a trained audio recognition model, a target object corresponding to the first audio data can be determined from multiple first objects. Since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, according to the display position of the target object, the subtitle information corresponding to the first audio data is displayed, which can facilitate users to understand the target object corresponding to the subtitle information and improve the user's video viewing experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic flow chart of a subtitle display method provided in an embodiment of the present application;

FIG2 is a schematic diagram of an image frame corresponding to audio data provided by an embodiment of the present application;

FIG3 is a schematic diagram of a subtitle display provided by an embodiment of the present application;

FIG4 is a schematic diagram of the structure of a subtitle display device provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

FIG6 is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated with each other are in an "or" relationship.

In response to the problems arising from the background technology, the embodiments of the present application provide a subtitle display method, device, electronic device and readable storage medium. During the playback of a target video, for a first image frame including multiple first objects, a target object corresponding to the first audio data can be determined from multiple first objects by using a trained audio recognition model. Since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, the subtitle information corresponding to the first audio data is displayed according to the display position of the target object, which can facilitate users to understand the target object corresponding to the subtitle information and improve the user's video viewing experience.

The following is a detailed description of the subtitle display method, device, electronic device and readable storage medium provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

FIG. 1 is a flow chart of a subtitle display method provided in an embodiment of the present application. The subtitle display method may include steps 110 to 120 .

Step 110 , during the playback of the target video, when it is detected that the first image frame of the target video includes a plurality of first objects, a target object corresponding to the first audio data is determined from the plurality of first objects according to an audio recognition model.

The first audio data corresponds to the first image frame, the audio recognition model is trained based on the second image frame and the second audio data corresponding to the second image frame, and the second image frame includes a single first object.

In the embodiment of the present application, the target video may be a TV series, a movie, a short video, a variety show, a video clip, etc., or a live broadcast, or a live broadcast playback, or other videos, which are not limited here. In order to facilitate users to appreciate the various languages included in the target video and understand the original sound in the target video, during the process of playing the target video, the subtitle information displayed on the playback interface may be obtained in real time during the video playback process, or may be the subtitle information cached in the electronic device after the electronic device plays the target video, which is not specifically limited here.

In some embodiments, there may be one first object or multiple first objects in the target video. For example, some frames in a film or TV series may include only one character, or some frames may include multiple characters at the same time. For another example, in a live broadcast, some frames may include one anchor, or some frames may include multiple anchors. The examples are not listed here.

In the embodiment of the present application, the first image frame is an image frame corresponding to when the target video includes two or more first objects, and the second image frame is an image frame corresponding to when the target video includes one first object.

In the embodiment of the present application, the target object corresponding to the audio data can be identified through the audio recognition model. In order to improve the reliability of the audio recognition model, in some embodiments, the second image frame and the second audio data corresponding to the second image frame can be used as training data. Since the second image frame only includes one first object, it can be considered that the audio data corresponding to the second image frame comes from the first object included in the second image.

Based on this, the electronic device can obtain multiple second image frames in the target video, and the audio data corresponding to each second image frame. Exemplarily, the electronic device can obtain audio feature data in the audio data corresponding to each second image frame, wherein the audio feature data may include, for example, word usage habits, timbre, pitch, and voiceprint information. The audio feature data of the audio data corresponding to each second image frame can be used as training data to train an audio recognition model.

In an embodiment of the present application, since the audio recognition model is trained based on the audio data of the target video itself and the first object in the image frame as training data, when the first image frame includes multiple first objects, when parsing the first audio data corresponding to the first image frame, it is possible to accurately determine the target object corresponding to the first audio data from the multiple first objects, so that the acquired subtitle information can accurately correspond to the target object among the multiple first objects.

As a specific embodiment, in order to improve the reliability of the audio recognition model, before parsing the first audio data corresponding to the first image frame according to the audio recognition model and determining the target object corresponding to the first audio data from multiple first objects, the embodiment of the present application may further include the following steps:

First, a training data set is obtained, wherein the training data set includes a second image frame and second audio data corresponding to the second image frame; next, a preset audio recognition network is trained according to the training data set to obtain an audio recognition model.

Specifically, the number of first objects included in each image frame in the target video can be confirmed based on image recognition technology. For example, the image recognition technology can identify whether the image frame includes a face. The image recognition technology can be, for example, a preset CNN algorithm, etc., which is not specifically limited here.

In some embodiments, since the audio feature data of the audio data corresponding to the second image frame may include, for example, word usage habits, timbre, pitch, and voiceprint information, the first object included in the second image frame may correspond one-to-one to the audio feature data corresponding to the second image frame, as training data for the preset audio recognition network. Since the training data comes from the target video itself, not only is the cost of acquiring the training data low, but the reliability of the training data is also high, thereby enabling effective training of the preset audio recognition network.

In order to improve the accuracy of distinguishing a target object from multiple first objects, in an embodiment of the present application, obtaining a training data set may specifically include the following steps:

According to the sound utterance time period of the first object in the target video, the target video is segmented to obtain a plurality of image frames corresponding to the audio data of the first object; from the plurality of image frames, a second image frame including a single first object and second audio data corresponding to the second image frame are obtained; the first object in the second image frame is associated with the second audio data to obtain a training data set.

Specifically, taking the target video as a movie as an example, when the audio data of the movie includes multiple lines of the first object, the sound of the first object in the target video may be, for example, a conversation between different first objects, or a monologue of the first object, etc., which is not specifically limited here.

When acquiring multiple image frames of the target video, the audio data of the entire movie can be divided according to the lines of each first object, in units of sentences, according to the start time point and the end time point of each line. Accordingly, the target video is segmented according to the start time point and the end time point of each line to obtain image frames corresponding to each line, and then the audio data of each line can be matched one by one with the image frame.

In some embodiments, according to a preset image recognition algorithm, the number of first objects in each image frame can be determined, and the image frame corresponding to two or more first objects is used as the first image frame, and the image frame corresponding to one first object is used as the second image frame.

FIG2 is a schematic diagram of an image frame corresponding to audio data provided by an embodiment of the present application. In combination with FIG2, according to the sounding time period of the first object, the audio data of the target video can be divided into audio 1 to audio 6 to obtain six sub-audio data. Correspondingly, based on the sounding time period of each first object, the target video can be divided into image frames 1 to 6 to obtain six image frames. Among them, image frame 1, image frame 2, and image frame 3 include first object A, first object B, and first object C, respectively, and image frame 1, image frame 2, and image frame 3 can be used as second image frames. Based on the fact that image frame 1, image frame 2, and image frame 3 correspond to audio 1, audio 2, and audio 3, respectively, audio 1 can be marked as the sound data of the first object A, audio 2 can be marked as the sound data of the first object B, and audio 3 can be marked as the sound data of the first object C.

Next, the first object A, the first object B, and the first object C can be associated with the audio 1, the audio 2, and the audio 3, respectively, to obtain a training data set for training a preset audio recognition network. In this way, not only can effective training data be obtained, but also because a large amount of labeled audio data does not need to be prepared in advance, the cost of training data is effectively reduced.

As a specific example, taking voiceprint as audio feature data, the preset audio recognition network can be, for example, a voiceprint recognition network based on a DNN algorithm, which is not specifically limited here.

According to the audio recognition model provided in the embodiment of the present application, it is possible to accurately distinguish that the current audio data belongs to a specific target object in the target video, and effective discrimination can be achieved even when multiple first objects speak at the same time.

As a specific embodiment, in combination with FIG. 2 , image frame 4, image frame 5, and image frame 6 all include two or more first objects, that is, image frame 4, image frame 5, and image frame 6 are all first image frames. Taking image frame 4 as an example, image frame 4 includes first object A, first object B, and first object C. According to the audio recognition model provided in the embodiment of the application, the audio 4 corresponding to image frame 4 can be analyzed, so as to determine that the audio 4 comes from the specific target object in image frame 4.

According to the embodiment of the present application, after determining the target object corresponding to the first audio data from the multiple first objects, step 120 may be performed next.

Step 120: display subtitle information corresponding to the first audio data according to the display position of the target object.

In some embodiments, subtitle information corresponding to the first audio data may be displayed next to the target object, so that the user can easily understand the specific target object speaking in the first image frame and the specific content said by the target object.

Exemplarily, through a preset audio recognition network, subtitle information corresponding to the first audio can be obtained. In order to facilitate the user to understand the target object corresponding to the audio data, the subtitles can be displayed near the target object, for example, they can be displayed next to the target object in the form of bubbles. Referring to FIG3, taking image frame 4 as an example, by parsing audio 4, it is recognized that audio 4 comes from the first object B in image frame 4. Combined with the subtitle display schematic diagram shown in FIG3, the subtitle information corresponding to audio 4 can be displayed next to the first object B in the form of bubbles.

As a specific example, when the first audio data includes multiple first objects making sounds at the same time, a speech separation algorithm can be used to separate the first audio data into multiple sub-audio data, and then the sub-audio data can be analyzed by the audio recognition model to determine the target object corresponding to each sub-audio data. If it is identified that multiple target objects have corresponding subtitle information, the subtitle information can be displayed next to each target object, which is not specifically limited here.

In some embodiments, the display mode of subtitles can be set according to the actual needs of the user, for example, the display color, font, size of subtitles, etc. of the subtitles, which are not specifically limited here.

In an embodiment of the present application, for a target video being played, when it is detected that a single first object is included in the second image frame of the target video, subtitle information corresponding to the second audio data is obtained; next, the subtitle information is displayed according to the display position of the single first object.

According to the embodiment of the present application, by displaying corresponding subtitle information according to the location of the first object, the user does not need to focus on the bottom of the screen, avoids missing the wonderful pictures or plots in the picture frame, and improves the user's experience of watching the video.

In some embodiments, by adding subtitles to each first object in the target video when playing the target video, the first object corresponding to each audio data can be accurately obtained. Next, the first object corresponding to the audio data and the subtitle information of the audio data can be cached in the electronic device, so that when the target video is played again, the subtitle information corresponding to each first object in the target video when it makes a sound can be quickly displayed, thereby improving the user experience.

It should be noted that the subtitle display method provided in the embodiment of the present application can be executed by a subtitle display device or a control module in the subtitle display device for executing the subtitle display method. In the embodiment of the present application, the subtitle display device executing the subtitle display method is taken as an example to illustrate the subtitle display device provided in the embodiment of the present application.

FIG4 is a schematic diagram of the structure of a subtitle display device provided in an embodiment of the present application. As shown in FIG4 , a subtitle display device 400 is applied to a bullet screen generating device. The subtitle display device 400 may include: a processing module 410 and a display module 420 .

The processing module 410 is used to determine, during the playback of the target video, a target object corresponding to the first audio data from the multiple first objects according to the audio recognition model when it is detected that the first image frame of the target video includes multiple first objects, wherein the first audio data corresponds to the first image frame;

The display module 420 is used to display subtitle information corresponding to the first audio data according to the display position of the target object.

The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

According to the embodiment of the present application, since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, according to the display position of the target object, the subtitle information corresponding to the first audio data is displayed, which can facilitate the user to understand the target object corresponding to the subtitle information and improve the user's video viewing experience.

In some embodiments, the apparatus further comprises:

The acquisition module is further used to acquire subtitle information corresponding to the second audio data when it is detected that the second image frame of the target video includes a single first object;

The display module 420 is further configured to display subtitle information according to the display position of the single first object.

According to the embodiment of the present application, by displaying corresponding subtitle information according to the location of the first object, the user does not need to focus on the bottom of the screen, avoids missing the wonderful pictures or plots in the picture frame, and improves the user's experience of watching the video.

In some embodiments, the acquisition module is used to acquire a training data set, the training data set including the second image frame and the second audio data corresponding to the second image frame;

The processing module 410 is further used to train a preset audio recognition network according to the training data set to obtain an audio recognition model.

In this way, since the training data comes from the target video itself, not only is the cost of obtaining the training data low, but the reliability of the training data is also high, thereby enabling effective training of the preset audio recognition network.

In some embodiments, the processing module 410 is further configured to segment the target video according to the sound utterance time period of the first object in the target video to obtain a plurality of image frames corresponding to the audio data of the first object;

The acquisition module is further used to acquire, from the plurality of image frames, a second image frame including a single first object and second audio data corresponding to the second image frame;

The processing module 410 is further configured to associate the first object in the second image frame with the second audio data to obtain a training data set.

In this way, not only can effective training data be obtained, but also the cost of training data is effectively reduced because there is no need to prepare a large amount of labeled audio data in advance.

The subtitle display device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

The subtitle display device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

The subtitle display device provided in the embodiment of the present application can implement each process implemented by the method embodiments of Figures 1 to 4, and will not be described again here to avoid repetition.

Optionally, as shown in FIG5 , an embodiment of the present application further provides an electronic device 500, comprising a processor 501, a memory 502, and a program or instruction stored in the memory 502 and executable on the processor 501. When the program or instruction is executed by the processor 501, each process of the above-mentioned subtitle display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

FIG. 6 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The electronic device structure shown in FIG6 does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently, which will not be described in detail here.

Processor 610 is configured to determine, during playback of a target video, when it is detected that a first image frame of the target video includes a plurality of first objects, according to an audio recognition model, a target object corresponding to first audio data from the plurality of first objects, wherein the first audio data corresponds to the first image frame;

A display unit 606, configured to display subtitle information corresponding to the first audio data according to a display position of the target object;

The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

According to the embodiment of the present application, since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, according to the display position of the target object, the subtitle information corresponding to the first audio data is displayed, which can facilitate the user to understand the target object corresponding to the subtitle information and improve the user's video viewing experience.

In some embodiments, the processor 610 is further configured to obtain subtitle information corresponding to the second audio data when it is detected that the second image frame of the target video includes a single first object;

The display unit 606 is further configured to display subtitle information according to the display position of the single first object.

According to the embodiment of the present application, by displaying corresponding subtitle information according to the location of the first object, the user does not need to focus on the bottom of the screen, avoids missing the wonderful pictures or plots in the picture frame, and improves the user's experience of watching the video.

In some embodiments, the processor 610 is configured to obtain a training data set, where the training data set includes a second image frame and second audio data corresponding to the second image frame;

The processor 610 is further configured to train a preset audio recognition network according to a training data set to obtain an audio recognition model.

In this way, since the training data comes from the target video itself, not only is the cost of obtaining the training data low, but the reliability of the training data is also high, thereby enabling effective training of the preset audio recognition network.

In some embodiments, the processor 610 is further configured to segment the target video according to a time period of a first object making a sound in the target video, to obtain a plurality of image frames corresponding to the audio data of the first object;

The processor 610 is further configured to obtain, from the plurality of image frames, a second image frame including a single first object and second audio data corresponding to the second image frame;

The processor 610 is further configured to associate the first object in the second image frame with the second audio data to obtain a training data set.

In this way, not only can effective training data be obtained, but also the cost of training data is effectively reduced because there is no need to prepare a large amount of labeled audio data in advance.

It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 609 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 610 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 610.

The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned subtitle display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned subtitle display method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims (4)

1. A subtitle display method, the method comprising:

In the process of playing a target video, under the condition that a plurality of first objects are included in a first image frame of the target video, dividing the target video according to a sounding time period of the first objects in the target video to obtain a plurality of image frames corresponding to audio data of the first objects;

Acquiring a second image frame comprising a single first object and second audio data corresponding to the second image frame from a plurality of image frames;

Establishing an association relation between the second audio data and a first object in the second image frame to obtain a training data set, wherein the training data set comprises the second image frame and second audio data corresponding to the second image frame;

Training a preset audio recognition network according to the training data set to obtain an audio recognition model;

determining a target object corresponding to first audio data from the plurality of first objects according to the audio recognition model, wherein the first audio data corresponds to the first image frame;

displaying caption information corresponding to the first audio data according to the display position of the target object;

Acquiring subtitle information corresponding to the second audio data under the condition that a single first object is included in a second image frame of the target video is detected;

And displaying the caption information according to the display position of the single first object.

2. A subtitle display apparatus, the apparatus comprising:

The processing module is used for dividing the target video according to the sounding time period of the first object in the target video under the condition that a plurality of first objects are included in the first image frame of the target video in the playing process of the target video, so as to obtain a plurality of image frames corresponding to the audio data of the first objects;

the acquisition module is also used for acquiring a second image frame comprising a single first object and second audio data corresponding to the second image frame from a plurality of image frames;

the processing module is further configured to correlate the first object in the second image frame with the second audio data to obtain a training data set, where the training data set includes the second image frame and second audio data corresponding to the second image frame; training a preset audio recognition network according to the training data set to obtain an audio recognition model;

The processing module is further configured to parse first audio data corresponding to the first image frame according to the audio recognition model, and determine a target object corresponding to the first audio data from the plurality of first objects, where the first audio data corresponds to the first image frame;

the display module is used for displaying subtitle information corresponding to the first audio data according to the display position of the target object;

The acquisition module is further configured to acquire subtitle information corresponding to the second audio data when it is detected that the second image frame of the target video includes a single first object;

the display module is further used for displaying the caption information according to the display position of the single first object.

3. An electronic device comprising a processor, a memory and a program or instruction stored on the memory and running on the processor, which when executed by the processor, implements the steps of the subtitle display method of claim 1.

4. A readable storage medium, wherein a program or instructions are stored on the readable storage medium, which when executed by a processor, implement the steps of the subtitle display method according to claim 1.