CN118830020A - Structured video documentation - Google Patents

Abstract

A method (500) includes: receiving a content feed (120) comprising audio data (122) corresponding to a speech utterance; and processing the content feed to generate a semantically rich, structured document (300). The structured document includes a transcription (310) of the speech utterance (123) and includes a plurality of words (123) each aligned with a corresponding audio segment (222) that indicates a time at which the word was recognized in the audio data. During playback of the content feed, the method further comprises: receiving a query from a user requesting information contained in a content feed (112); and processing the query and the structured document by the large language model (180) to generate a response to the query (182). The response conveys the requested information contained in the content feed. The method further comprises the steps of: a response to the query is provided for output from a user device associated with the user (102).

Description

Structured video documentation

Technical Field

The present disclosure relates to structured video documents.

Background Art

While video is a common way for users to consume entertainment, news, and educational content, it remains challenging for users to use video as an information medium due to limitations imposed by the ability to search and call up the content of a video. For information-based tasks, users typically interact with a timeline-based video player user interface to scrub through a video forward/backward to locate specific content that may be of interest. To some extent, the ability to generate transcripts/narratives for dialogues in a video has improved the ability to search for content in a video by allowing users to enter keyword searches to locate relevant content in the transcripts/narratives. However, these user interfaces that use transcripts/narratives to search for content lack the ability to semantically understand queries that are spoken (or typed) for specific content in a video—not to mention the ability to use semantically relevant information to implement queries.

Summary of the invention

One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising: receiving a content feed including audio data corresponding to a speech utterance; and processing the content feed to generate a semantically rich structured document. The structured document includes a transcription of the speech utterance and includes a plurality of words that are each aligned with a corresponding audio segment of the audio data, the corresponding audio segment indicating the time at which the word was recognized in the audio data. During playback of the content feed, the operations also include: receiving a query from a user requesting information contained in the content feed; and processing the query and the structured document by a large language model to generate a response to the query. Here, the response conveys the requested information contained in the content feed. The operations also include: providing a response to the query for output from a user device associated with the user.

Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation also includes: extracting a transcribed segment including the requested information conveyed by the response to the query, the transcribed segment being bounded by a start word and an end word; identifying the starting audio segment of the audio data as the corresponding audio segment of the audio data aligned with the starting word that bounds the transcribed segment; and identifying the ending audio segment of the audio data as the corresponding audio segment of the audio data aligned with the ending word that bounds the transcribed segment. In these implementations, providing a response to the query includes replaying back the audio data from a user device associated with the user from the starting audio segment of the audio data to the ending audio segment of the audio data. The content feed may further include image data, the image data including multiple image frames, wherein the operation further includes: pausing the playback of multiple image frames of the image data while replaying the audio data from the starting audio segment of the audio data to the ending audio segment of the audio data.

In some examples, the content feed further includes image data, the image data including a plurality of image frames, and the semantically enriched structured document further includes creator-provided text identified in one or more of the plurality of image frames. Here, the creator-provided text is aligned with a corresponding audio segment of the audio data to indicate a time when the creator-provided text was identified in the one or more image frames. In these examples, processing the content feed to generate the semantically enriched structured document may further include annotating a transcription of the speech utterance with the creator-provided text by inserting the creator-provided text between a pair of adjacent words in the transcription based on a corresponding audio segment of the audio data aligned with the creator-provided text identified in the one or more image frames.

The response to the query may include a text response that conveys the requested information as a coherent, focused response to the query. In some implementations, the operation also includes: performing text-to-speech conversion on the text response to generate a synthesized speech representation of the response to the query, wherein providing the response to the query for output from a user device includes: audibly outputting the synthesized speech representation of the response to the query from the user device. In these implementations, the operation may further include: pausing playback of the content feed while the synthesized speech representation of the response to the query is audibly output from the user device. Further, the text response to the query may further include one or more references to source materials related to the requested information.

In some examples, the large language model includes a pre-trained large language model, and few-shot learning is performed using a structured document as a context for a query to generate a response to the query. The query may include a question in natural language, and the response to the query may include a natural language response to the question.

In some implementations, processing a content feed to generate a semantically rich structured document includes: segmenting audio data into a plurality of audio segments; performing speaker classification on the plurality of audio segments to predict a classification result including a corresponding speaker label assigned to each audio segment; and indexing a transcription of the speech utterance using the corresponding speaker label assigned to each audio segment segmented from the audio data.

Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: receiving a content feed comprising audio data corresponding to a speech utterance; and processing the content feed to generate a semantically rich structured document. The structured document comprises a transcription of the speech utterance and comprises a plurality of words each aligned with a corresponding audio segment of the audio data, the corresponding audio segment indicating the time when the word was recognized in the audio data. During playback of the content feed, the operation also comprises: receiving a query from a user requesting information contained in the content feed; and processing the query and the structured document by a large language model to generate a response to the query. Here, the response conveys the requested information contained in the content feed. The operation also comprises: providing a response to the query for output from a user device associated with the user.

This aspect may include one or more of the following optional features. In some implementations, the operation also includes: extracting a transcribed segment including the requested information conveyed by the response to the query, the transcribed segment being bounded by a start word and an end word; identifying a starting audio segment of the audio data as a corresponding audio segment of the audio data aligned with the start word that bounds the transcribed segment; and identifying an ending audio segment of the audio data as a corresponding audio segment of the audio data aligned with the end word that bounds the transcribed segment. In these implementations, providing a response to the query includes replaying the audio data from a user device associated with the user from the starting audio segment of the audio data to the ending audio segment of the audio data. The content feed may further include image data, the image data including a plurality of image frames, wherein the operation further includes: pausing playback of the plurality of image frames of the image data while replaying the audio data from the starting audio segment of the audio data to the ending audio segment of the audio data.

In some examples, the content feed further includes image data, the image data including a plurality of image frames, and the semantically enriched structured document further includes creator-provided text identified in one or more of the plurality of image frames. Here, the creator-provided text is aligned with a corresponding audio segment of the audio data to indicate a time when the creator-provided text was identified in the one or more image frames. In these examples, processing the content feed to generate the semantically enriched structured document may further include annotating a transcription of the speech utterance with the creator-provided text by inserting the creator-provided text between a pair of adjacent words in the transcription based on a corresponding audio segment of the audio data aligned with the creator-provided text identified in the one or more image frames.

The response to the query may include a text response that conveys the requested information as a coherent, focused response to the query. In some implementations, the operation also includes: performing text-to-speech conversion on the text response to generate a synthesized speech representation of the response to the query, wherein providing the response to the query for output from a user device includes: audibly outputting the synthesized speech representation of the response to the query from the user device. In these implementations, the operation may further include: pausing playback of the content feed while the synthesized speech representation of the response to the query is audibly output from the user device. Further, the text response to the query may further include one or more references to source materials related to the requested information.

In some examples, the large language model includes a pre-trained large language model, and few-shot learning is performed using a structured document as a context for a query to generate a response to the query. The query may include a question in natural language, and the response to the query may include a natural language response to the question.

In some implementations, processing a content feed to generate a semantically rich structured document includes: segmenting audio data into a plurality of audio segments; performing speaker classification on the plurality of audio segments to predict a classification result including a corresponding speaker label assigned to each audio segment; and indexing a transcription of the speech utterance using the corresponding speaker label assigned to each audio segment segmented from the audio data.

The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a schematic diagram of an example system for allowing a user to interact with a video using a structured document generated for the video.

2 is a schematic diagram of an example document structurer for generating a structured document from audio data and image data of an audiovisual feed.

3 is a diagram of an example structured document including a transcription, author-provided text, and annotated transcription.

4 is a diagram of an example video interface that presents information from a structured document generated for a video while playing back the video.

5 is a flow diagram of an example arrangement of operations of a method for interacting with a content feed using a structured document during playback of the content feed.

6 is a schematic diagram of an example computing device that may be used to implement the systems and methods described herein.

Like reference numbers in the various drawings indicate like elements.

DETAILED DESCRIPTION

The video player adopted by media playback application and web browser allows users to provide commands for controlling the playback of videos. For example, users can play/pause/stop videos, and scan forward/backward via dedicated buttons/commands or via dragging through the video timeline. The latest progress in automatic speech recognition (ASR) has made it possible for users to provide these video playback commands by voice. The video timeline feature enables users to preview frame-by-frame visual content, but requires users to drag forward and backward through the timeline multiple times to locate the content of interest. For example, if a user is watching a cooking instruction video for preparing a specific recipe, wherein the actor/instructor/participant (also referred to as "speaker") speaks a list of ingredients and its corresponding proportions, in the case where the user does not have time to internalize a required proportion for one of the ingredients, the user will have to manually drag backward in the video to repeat the playback of the instructor saying the list of ingredients. Obviously, dragging through the timeline for users to engage in locating the content of interest is an inefficient and time-consuming process. Furthermore, interactive timeline searching is limited to the frame-by-frame visual content of the video and does not reveal audio content or high-level content such as plot/scene description or theme.

Some video player user interfaces utilize text transcriptions and transcripts of the audio content in the video to support keyword searches entered by the user. In the above example where the user is watching an instructional cooking video, while the speaker is speaking the proportions of cumin, the user can enter a (verbal or text) keyword search for "how much cumin?" to ascertain the proportions of cumin from the transcription of the audio content. However, assuming that the instructor/actor merely referenced the list of ingredients by name without specifying the proportions of each ingredient required for the recipe, and instead presented/superimposed a creator-provided graphic that visually conveyed the proportions of each ingredient by the creator of the instructional cooking video, the user would not be able to ascertain the information he/she is looking for via a keyword search because such information is missing from the transcription.

Additionally, information extracted from transcribed and narrated speech in response to keyword searches is often time consuming to read and difficult to locate relevant content because transcribed and narrated speech tend to contain speech-specific disfluencies and redundancies. Since the transcription presents only long segments of text and the narration contains only sequences of short phrases, this lack of structured organization limits the user's ability to navigate to specific topics in the video or to ascertain any type of summary of the video's content.

Some inherent defects in the prior art for searching related content in a video can be solved by allowing the creator of the video to embed structured text in the spirit of creating a navigable representation of the video to realize the purpose of searching for the content queried by the user/viewer. In addition to transcription and commentary, the creator can also embed a structured text document that conveys key themes, chapter titles, plot summaries, and summaries for different segments of the video into the video. Although the structured text document provided by the creator can effectively allow the user/viewer to locate related content in the video to a certain extent, the resources and expenses required for creating related structured text documents and embedding them into the video make this task difficult for the vast majority of creators to perform. In addition, even if the creator is willing to create a structured text document for their video content, the content returned in response to the user's search is only as good as the structured text document that the creator chooses to embed into the video. That is, it is simply an impossible feat for the creator to anticipate all conceivable types of content that may be the subject of the user's search so as to be included in the structured text document. By the same concept, embedding of creator-provided structured text documents into videos also fails to provide a truly interactive experience for users/viewers when entering queries to locate content of interest in the video, since the creator-provided structured text documents cannot be leveraged in a unified way for semantic interpretation of queries.

The implementation of this article relates to automatically generating semantically rich structured documents for content feeds (i.e., videos) to achieve semantic interpretation of queries requesting information contained in content feeds. Referring to Figure 1, system 100 includes a user 2 who views a content feed 120 played back on a computing/user device 10 through a media player application 150. The media player application 150 can be a stand-alone application executed on the user device 10 or a web-based application accessed via a web browser. In the example shown, the content feed 120 includes a recorded instructional cooking video played on the computing device 10 for user 2 to watch and interact with. Although the examples of this article depict the content feed 120 as an audio-visual (AV) feed (e.g., video) including audio data 122 (e.g., audio content, audio signal, or audio stream) and image data 124 (e.g., image content or video content), the content feed 120 can be a pure audio feed including only audio data 122, such as, but not limited to, a podcast episode or an audio book. For simplicity, content feed 120 may be referred to herein interchangeably as video, AV signal, AV feed, or simply AV data, unless otherwise specified.

The system 100 also includes a remote system 130 that communicates with the computing device 10 via a network 120. The remote system 130 may be a distributed system with scalable/elastic resources (e.g., a cloud computing environment or a storage abstraction). The resources include computing resources 134 (e.g., data processing hardware) and/or storage resources 136 (e.g., memory hardware). In some implementations, the remote system 130 hosts (e.g., on computing resources) a media player application 150 to coordinate playback of the content feed 120 on the computing device 10, generates semantically rich structured documents 300 for the content feed 120, and uses the structured documents 300 to enable the user 10 to interact with the content feed 120 via the computing device 10 by issuing queries 112 that request information contained in the content feed during playback of the content feed 120. For example, the data processing hardware 134 of the remote system 130 may execute instructions stored on the memory hardware 136 of the remote system 130 for executing the application 150. Additionally or alternatively, media player application 150 may be executed on computing device 10 associated with user 2. For example, data processing hardware 12 of computing device 10 may execute instructions stored on memory hardware 14 of computing device 10 for executing application 150. Some examples of data processing hardware 12 include a central processing unit (CPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

Computing device 10 includes, or is in communication with, a display 11 capable of displaying a video interface 400 for presenting image data 124 and a speaker 18 for audible output of audio data 122. Audio data 122 may correspond to voice utterances 123 spoken by an actor, instructor, narrator, meeting participant, presenter, or other individual recorded in video 120. Some examples of computing device 10 include computers, laptops, mobile computing devices, smart televisions, monitors, smart devices (e.g., smart speakers, smart displays, smart appliances), wearable devices, etc. In the example shown, content feed 120 includes a recorded instructional cooking video that is played on computing device 10 for user 2 to view and interact with. The implementation of the present invention involves a media player application 150 that provides an interactive experience to a user 2 during playback of an instructional cooking video 120, wherein the media player application allows the user 2 to issue a natural language query 112 requesting information contained in the video 120, whereby the application 150 uses a semantically rich structured document 300 generated for the video 120 to retrieve the requested information and provide a response 182 containing the requested information to the user 2.

User 2 may issue query 112 as a spoken query 112 that is captured in streaming audio by microphone 16 in communication with computing device 10, and application 150 (or another application) may perform speech recognition to convert spoken query 112 into a corresponding textual representation of query 112. Alternatively, user 2 may also have the ability to enter query 112 via input device 20, which may include a physical keyboard in communication with computing device 10 or a virtual keyboard presented for display in video interface 400. Input device 20 may also include a mouse, stylus, or graphical user interface that permits a user to enter query 112 requesting information about an object displayed in video data 124 (e.g., a word in a closed caption, a word/phrase in author-provided text, an entity depicted in the context of the video) by selecting or hovering over the object.

For example, an instructional cooking video 120 is playing a segment in which an actor is mixing a list of ingredients for making a popular Thai seafood curry dish called Haw Mok Talay. If user 2 is unable to internalize the recently communicated ratio of cumin during playback of video 120, the user can issue a query 112 "How much cumin?" to find out the ratio of cumin. Without the user having to manually drag back in video 120 to locate the segment when the ratio of cumin is communicated, application 150 can retrieve the ratio of cumin (i.e., the requested information) from a semantically rich structured document 300. For example, assuming that the actor explicitly says the ratio of cumin in voice utterance 123, application 150 can retrieve the ratio of cumin from transcription 310 of audio data 122. Among other types of information, structured document 300 can also include creator-provided text 320 identified in image data 124. For example, the creator of the video 120 may briefly display creator-provided text 320 in the image data 124 that specifies that the recipe calls for half a teaspoon of cumin. Thus, the application 150 may retrieve the ratio of cumin from the creator-provided text 320 included in the structured document 300, and provide a response 182 to the user's query 112 that communicates that half a teaspoon of cumin is called for, regardless of whether the actor explicitly says the ratio in the voice utterance 123.

The media player application 150 includes a document structurer 200, a large language model 180, and an output module 190. The document structurer 200 is configured to receive/ingest and process the audio-visual feed 120 to generate a semantically rich structured document 300. Notably, the document structurer 200 can automatically generate the structured document 300 for the ingested audio-visual feed 120 without requiring the creator of the audio-visual feed 120 to provide any structured text or otherwise require the creator to contribute in the creation of the structured document 300. Thus, the document structurer 200 can ingest any new or existing content feed 120 and immediately generate the structured document 300 without any input from the creator of the feed 120.

The audiovisual feed 120 ingested by the document structurer 200 includes audio data 122 and image data 124. The audio data 122 may represent a speech utterance 123, and the image data 124 may include a plurality of image frames 125, 125a-n (FIG. 2). As discussed above, the content feed 120 may include a pure audio feed 120 that includes only the audio data 122. The structured document 300 includes a transcription 310 of the speech utterance 123. As will be described in more detail below with reference to FIG. 2, the transcription 310 includes a plurality of words, and the structured document 300 aligns each word in the transcription 310 with a corresponding audio segment 222 (FIG. 2) of the audio data 122, the corresponding audio segment indicating the time at which the word was recognized in the audio data 122. That is, the structured document 300 includes a timestamp for each word in the transcription 310.

In some implementations, the document structurer 200 also processes the image data 124 to determine whether any author-provided text 320 is recognized in the image data 124. In these implementations, the structured document 300 generated by the document structurer 200 will also include any author-provided text 320 recognized in one or more image frames 125a-n (FIG. 2) of the image data 124. The document structurer 200 may use techniques such as object character recognition to recognize any author-provided text 320 in each image frame 125. As used herein, author-provided text 320 may include text graphics (any combination of letters, words, or other symbols) that the author of the video superimposed on the scene depicted in the image frame in order to convey relevant content to the user/viewer 2. Author-provided text 320 may also include any text recognized in the actual scene depicted in the image frame. The structured document 300 may align any recognized author-provided text 320 with a corresponding audio segment 222 ( FIG. 2 ) of the audio data 122 to indicate when the author-provided text 320 was recognized in one or more image frames 125 .

The document structurer 200 may also additionally perform other processing techniques on the ingested audio-visual feed 120, such as but not limited to speaker diarization, summarization, and formatting, and save the results of these processing techniques in a structured document 300. Speaker diarization answers the question "who is speaking when" and has a variety of applications, including multimedia information retrieval, speaker turn analysis, audio processing, and automatic transcription of conversational speech, among others. The document structurer 200 may utilize a text generation model that consumes the transcription 310 and/or the author-provided text 320 and outputs key topics or corresponding summaries of one or more different segments of the audio-visual feed 120. Formatting may identify different chapters/scenes in the audio-visual feed 120.

2 shows an example of a document structurer 200 including a classification module 220, an automated speech recognition (ASR) module 230, an object character recognition (OCR) module 240, and a generator 250. The application 150 executes the ASR module 230 to generate a transcription 310 (also referred to as a transcription) of one or more speech utterances 123 spoken by a speaker (e.g., an actor/participant) in a content feed 120 (e.g., an audio-visual signal including audio data 122 and video data 124 or an audio-only signal including only audio data 122).

The classification module 220 is configured to receive audio data 122 corresponding to an utterance 123 of a speaker from the content feed 120 (and optionally image data 124 representing the face of the speaker), segment the audio data 122 into a plurality of segments 222, 222a-n (e.g., fixed-length segments or variable-length segments), and generate a classification result 224 based on the audio data 122 (and optionally the image data 124) using a probabilistic model (e.g., a probabilistic generative model), the classification result including a corresponding speaker label 226 assigned to each segment 222. In other words, the classification module 220 includes a series of speaker identification tasks with short utterances (e.g., segments 222), and determines whether two segments 222 of a given conversation are spoken by the same speaker. At the same time, the classification module 220 can simultaneously perform a facial tracking routine to identify which participant is speaking during which segment 222 to further optimize the speaker identification. The classification module 220 is then configured to repeat the process for all segments 222 of the conversation. Here, the classification results 224 provide timestamped speaker labels 226 , 226 a - n for the received audio data 122 , which can identify not only who is speaking during a given segment 222 , but also when speaker changes occur between adjacent segments 222 .

The ASR module 230 is configured to receive audio data 122 corresponding to the utterance 123 (and optionally image data 124 representing the face of the speaker when the utterance 123 is spoken). The ASR module 230 transcribes the audio data 122 into corresponding ASR results 232. Here, the ASR results 232 refer to a text transcription (e.g., transcription 310) or multiple candidate text transcriptions of the audio data 122. In some examples, the ASR module 230 communicates with the classification module 220 to improve speech recognition based on the utterance 123 using the classification results 224 associated with the audio data 122. For example, the ASR module 230 can apply different speech recognition models (e.g., language models, prosody models) to different speakers identified from the classification results 224. Additionally or alternatively, the ASR module 230 and/or the classification module 220 (or some other component of the application 150) can index the transcription 310 of the audio data 122 using the predicted timestamped speaker labels 226 for each segment 222 obtained from the segmentation results 224. As shown in FIG. 2 , the transcription 310 for the content feed 120 can be indexed by speaker to associate portions of the transcription 202 with corresponding speakers in order to identify what each speaker said.

In some implementations, the document structurer 200 receives a transcript for the spoken utterance 123 that was previously generated by another application or provided by an author of the content feed 120. The transcript can be time-stamped/aligned with the audio segment 222 of the audio data 122 to indicate the time when the transcript for the utterance 123 was spoken. The transcript can be used as a transcription 310 without the ASR module 230 processing the audio data 122, or the transcript can be used in conjunction with or to improve the recognition result 232 of the transcription 310 generated by processing the audio data 122. In some examples, when the transcript does not include punctuation, the document structurer 200 adds punctuation to the previously generated transcript for improving the accuracy of the response 182 generated by the large language model 180.

The transcription 310 for the utterance 123 included in the structured document 300 also includes alignment information 315. The alignment information 315 provides an alignment between each word 312 (FIG. 3) of the plurality of words 312, 312a-n (FIG. 3) of the transcription 310 and a corresponding audio segment 222 of the audio data 122, the corresponding audio segment indicating a time at which the corresponding word was recognized.

The OCR module 240 is configured to recognize any creator-provided text 320 that may be present in one or more image frames 125a-n of the image data 124. The OCR module 240 may include an OCR machine learning model (e.g., recognizer) 244 that is trained to recognize any creator-provided text 320 in each image frame 125. As used herein, the creator-provided text 320 may include text graphics (any combination of letters, words, or other symbols) that the creator of the video superimposes on the scene depicted in the image frame in order to convey relevant content to the user/viewer 2. The creator-provided text 320 may also include any text recognized in the actual scene depicted in the image frame 125. In some examples, the OCR module 240 includes an OCR data store 242 that the OCR machine learning model 244 can access to recognize specific fonts, symbols, or text patterns. The creator-provided text 320 recognized in one or more image frames 125a-n may further include corresponding alignment information 322. Here, the alignment information 322 provides an alignment between any recognized author-provided text 320 and the corresponding audio segment 222 ( FIG. 2 ) of the audio data 122 to indicate the time at which the author-provided text 320 was recognized in one or more image frames 125 .

In some scenarios, the ASR module 230 uses the recognized author-provided text 320 and the corresponding alignment information 322 to improve the accuracy of the transcription 310. Referring to the above example where the content feed 120 includes an instructional cooking video, the ASR module 230 can generate a recognition result 232 that misrecognizes the name of the Thai dish "Haw Mok Talay" as "Hamook Taley". Similarly, the previously generated commentary can misrecognize the name of the dish. Some of the author-provided text 320 recognized in the image data 320 can include the phrase "Haw Mok Talay". In some cases, the correct spelling (“Haw Mok Talay”) may be a lower confidence hypothesis in the list of candidate hypotheses included in the speech recognition results 232, whereby a match with the phrase “Haw Mok Talay” present in the recognized author-provided text 320 increases the confidence of the candidate hypothesis for “Haw Mok Talay” in the recognition results 232, causing the phrase to ultimately be selected for inclusion in the transcription 310.

In some implementations, the generator 250 receives the transcription 310 recognized in one or more image frames 125, the author-provided text 320, and the corresponding alignment information 315, 322, and generates the structured document 300 by annotating the transcription 310 of the speech utterance 123 with the author-provided text 320. In these implementations, the alignment information 315, 322 can show the likelihood of which parts of the speech utterance 123 are related to the author-provided text 320. For example, the generator 250 can insert the author-provided text 320 between a pair of adjacent words in the transcription 310 based on the corresponding audio segment 222 of the audio data 120 that is aligned with the author-provided text 320 recognized in the one or more image frames.

FIG3 shows an example semantically enriched structured document 300 generated by the document structurer 200 of FIGS. 1 and 2 for an audiovisual feed 120 corresponding to an instructional cooking video. The transcription 310 is associated with an utterance 123 spoken by a speaker discussing some of the steps of making a satay marinade for the Thai dish "Haw Mok Talay". The transcription 310 includes a plurality of words 312, 312a-n, and corresponding alignment information 315 provides an alignment between each word 312 in the plurality of words 312a-n and a corresponding audio segment 222 of the audio data 122, the corresponding audio segment indicating the time when the corresponding word 312 was recognized. The author-provided text 320 includes "1.5 tsp coriander" and "0.5 tsp cumin" indicating the respective proportions of coriander and cumin required for the satay marinade. Notably, since the speaker never uttered any utterances 123 conveying these proportions, they are not included in the transcription 310. The corresponding alignment information 325 provides an alignment between the author-provided text 320 and the corresponding audio segments 222 (FIG. 2) of the audio data 122 to indicate when the author-provided text 320 was recognized in one or more image frames 125.

In the example shown, the structured document 300 also includes an annotated transcription 330 that uses the alignment information 315, 325 to insert the creator-provided text 320 between a pair of adjacent words (e.g., "anything" and "Coriander") in the transcription 310 based on the corresponding audio segment 222 of the audio data 120 that is aligned with the creator-provided text 320 recognized in the one or more image frames. Here, the annotated transcription 330 includes the creator-provided text 320 inserted into the relevant position of the transcription 310.

Referring back to FIG. 1 , in some implementations, during playback of a content feed (e.g., an audiovisual feed) 120, a large language model 180 is configured to receive as input a semantically rich structured document 300 and a query 112 issued by a user 2, and generate as output a response 182 to the query 112 that communicates requested information contained in the content feed 120. In some examples, the query 112 includes a question in natural language, and the response 182 to the query 112 includes a natural language response that provides an answer to the question. For example, the response 182 to the query 112 may include a text response generated by the large language model 180 that communicates the requested information as a coherent, focused response to the query 112. In some examples, the large language model 180 can further enhance the coherent/focused response 182 to the query 112 by citing source material to highlight the authority of the information contained in the response 182. That is, the text response 182 to the query 112 may include one or more references to source material related to the requested information, such as links to entities mentioned in the response 182 that may direct the user 2 to additional information. In addition to generating text to provide a natural language response/answer 182 to the natural language query 112, the large language model 182 may perform other generative tasks such as generating natural language text that summarizes one or more portions of the structured document 300.

The large language model 180 may include a pre-trained large language model 180, which is pre-trained based on general world knowledge using one or more generative tasks (i.e., multi-task learning) to learn efficient context representations. Therefore, the large language model 180 may include a multi-task unified model (MUM). The pre-trained large language model 180 may be based on a Transformer or Conformer model, or other encoding/decoding architectures with a multi-head attention mechanism. For example, the pre-trained large language model may include an encoding branch for encoding a structured document 300, another encoding branch for encoding a query 112, and a shared decoder that receives two encodings to retrieve/generate responses to answer queries. It is worth noting that the Transformer/Conformer model can be effectively parallelized for training a large-scale language model, which is proven to generalize better than a language model based on an autoregressive neural network architecture such as a recurrent neural network model, and achieve significantly better performance. The pre-trained neural network model 180 may include more than one billion parameters, and may exceed an upper limit of one trillion parameters.

Implementations herein involve a pre-trained large language model 180 that performs few-shot learning, which uses a structured document 300 as context to generate a response 182 to a query 180. That is, the few-shot learning can fine-tune the parameters of the pre-trained large language model 180 so that the language model 180 can be applied to the downstream task of retrieving relevant information contained in an audio-visual source in response to a query 112 issued by a user 2. The use of few-shot learning is particularly useful for tasks where limited training data is available, because the language model 180 is able to generalize well based on the structured document 300 provided with labeled examples to improve the retrieval of relevant data (e.g., information contained in the audio-visual feed 120 that the user 2 is currently watching). Through few-shot learning, the query 112 and the structured document 300 are provided as an output pair to the pre-trained large language model 300, such that the structured document is labeled as being related in some way to generating a response 182 as an output. The large language model 180 is also capable of performing zero-shot learning tasks, where the language model 180 can default to the language model's understanding of the world when generating a response 182 to a query 110 .

After the large language model 180 generates the response 182, the output module 190 is configured to provide the response 182 for output from the user computing device. The output module 190 may include any combination of a playback settings controller 190a, a user interface (UI) generator 190b, and a text-to-speech (TTS) system 190c. Continuing with the example in which the audio-visual feed 120 includes an instructional cooking video, upon realizing that details spoken by the actor in the video about how the coriander seeds used in the recipe were toasted are missing, the user 2 may provide the query 112 "What, how were they toasted?" The response 182 may include the answer "They were toasted in a dry sauté pan." In some examples, the output module 190 receives the response 182 and the structured document 300 as input and extracts a segment of the transcription 310 (and/or a segment of the annotated transcription 330 ( FIG. 3 )) that includes the requested information conveyed by the response to the query 112. For example, the segment extracted from the transcription 310 may include “You want to toast them in a dry sauté pan,” where the segment is bounded by the beginning word “You” and the ending word “pan.” Thus, the output module 190 may then both identify the starting audio segment 222 ( FIG. 2 ) of the audio data 122 as the corresponding audio segment that is aligned with the beginning word that bounds the segment of the transcription and identify the ending audio segment 222 of the audio data 122 as the corresponding audio segment that is aligned with the ending word that bounds the segment of the transcription 310. Using the identified starting and ending audio segments 222, the output module 190 can direct the playback settings controller 190a to replay the audio data 122 from the starting audio segment to the ending audio segment as an audible output from the speaker 18, so that the speech 123 "You want to toast them in a dry sauté pan" is replayed to convey the response 182 to the query 110. Notably, the playback settings controller 190a can pause the playback of multiple image frames of the image data 125 while replaying the associated audio data 120. In some examples, the controller 190a pauses the playback of the audio-video feed 110 in response to receiving the query 110.

In some additional examples, the output module 190 directs the TTS system 190c to perform a TTS transformation on the text response 182 output from the large language model 180 to generate a synthesized speech representation of the response 182 to the query 112. The output module 190 can use the TTS system 190c in scenarios when the requested information conveyed by the response to the query 112 is not present in the transcription 310, and thus is never conveyed in the spoken utterance 123. Thus, there is no opportunity to replay any portion of the audio data 122 to convey the requested information conveyed by the response 182. For example, and referring to FIG. 3, the response 182 to the query 112 "How much cumin?" can be determined solely by the large language model 180 from the author-provided text 320 as evidenced by the annotated transcription 330. In this example, the output module 190 may receive the text response 182 "half a teaspoon of cumin seeds" generated by the large language model 180 as an answer to the query 112, and direct the TTS system 190c to perform text-to-speech conversion on the text response 182 to generate a synthesized speech representation of the response 182 to the query 112. Accordingly, the media player application 150 may audibly output the synthesized speech representation from the speaker 18 of the computing device 10. Notably, the playback settings controller 190a may also completely pause playback of the audio-visual feed 120 when the synthesized speech representation of the response 182 "half a teaspoon of cumin seeds" is audibly output from the speaker 18 of the computing device 10. In some examples, the controller 190a pauses playback of the audio-visual feed 110 in response to receiving the query 110.

Additionally, the output module 190 can direct the UI generator 190b to generate a graphic of the text response 182 and present the graphic of the text response 182 in the video interface 400 displayed on the display 11 of the computing device 10 during the playback of the audio-visual feed 120. Here, the user 2 can simply read the graphic of the text response 182 presented in the video interface 400 while watching the video 120. The text response 182 may include one or more references to source materials related to the requested information. For example, the graphic of the text response 182 presented in the video interface 400 may provide a hyperlink to the reference to the source material related to the requested information. The user 2 can simply hover (e.g., by a mouse) over or touch a word of interest in the text response presented in the video interface 400 to view additional information or be directed to another source, such as a web page.

FIG. 4 provides an example video interface 400 displayed by the media player application 150 on the display 11 of the computing device 10 during playback of the audio-visual feed 120. In this example, the media player application 150 also displays information from a semantically enriched structured document 300 generated for the audio-visual feed 120 in the video interface 400 to allow the user 2 to interact with the structured document 300 during playback of the audio-visual feed 120. For example, a transcription 310 of an utterance spoken by two different speakers may be displayed, along with corresponding speaker tags 204 indicating which portions of the transcription 310 were spoken by each speaker. The structured document 300 may further provide multimodal interactions, such as hyperlinks added to specific terms or entities narrated in the transcription 310 that may be relevant to the user 2. For example, additional information about the term "cryptocurrency" may be explored by the user 2 via selecting the term or hovering the mouse over the term. Here, the video interface 400 may be populated with a definition of cryptocurrency or a snippet from a Wikipedia page about cryptocurrency.

The structured document 300 may further provide summaries 410 of relevant chapters/sections/scenes of the audio-visual feed 120 for presentation in the video interface 400. Here, the summaries 410 may be generated by the large language model 180 of FIG1 based on information extracted from the transcription 310, the author-provided text 320, and/or the annotated transcription 320. The user 2 may select one of the summaries 410, and the video player may advance to that portion of the video.

The video interface 400 of the media player application 150 also provides playback settings controls 450 that allow the user 2 to control the playback of the audio-visual feed 450. For example, the playback settings controls 450 may include buttons for play, forward/backward scanning, pause, and a video timeline that the user 2 can manipulate to scrub through the video.

5 provides a flow diagram of an example arrangement of operations of a method 500 for interacting with a content feed 120 using a structured document 300 during playback of the content feed 120. At operation 502, the method 500 includes receiving a content feed 120 including audio data 122. The audio data 120 corresponds to a speech utterance 123. The content feed 120 may include an audiovisual feed that additionally includes image data 124 that includes a plurality of image frames 125a-n.

At operation 504, the method 500 includes processing the content feed 120 to generate a semantically enriched structured document 300. Here, the structured document 300 includes a transcription 310 of the speech utterance 123. The transcription 310 may include a plurality of words 312 that are each aligned with a corresponding audio segment 222 of the audio data 122, the corresponding audio segment indicating the time at which the word 312 was recognized in the audio data 122.

At operation 506, during playback of the content feed 120, the method 500 includes receiving a query 112 from user 2 requesting information contained in the content feed. At operation 508, during playback of the content feed 120, the method 500 includes processing the query 112 and the structured document 300 by the large language model 180 to generate a response 182 to the query 112. Here, the response 182 conveys the requested information contained in the content feed 120. At operation 510, the method includes providing the response 182 to the query 112 for output from a user device 10 associated with user 2.

A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and/or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

FIG6 is a schematic diagram of an example computing device 600 that may be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and/or claimed in this document.

The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface/controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface/controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input/output device (such as a display 680 coupled to the high-speed interface 640). In other implementations, multiple processors and/or multiple buses can be used as appropriate along with multiple memories and multiple memory types. Moreover, multiple computing devices 600 can be connected, each of which provides a portion of the necessary operations (for example, as a server library, a blade server group, or a multi-processor system).

The memory 620 stores information non-transitorily within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transitory memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 (which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)) can be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

As shown, computing device 600 may be implemented in a variety of different forms. For example, it may be implemented as a standard server 600a or multiple times as a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

Various implementations of the systems and techniques described herein can be implemented in digital electronic and/or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and/or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and/or object-oriented programming languages and/or in assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and/or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and/or data to a programmable processor.

The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or a touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the appended claims.

Claims (24)

1. A computer-implemented method (500), the computer-implemented method (500) when executed on data processing hardware (134) causing the data processing hardware (134) to perform operations comprising:

Receiving a content feed (120) comprising audio data (122), the audio data (122) corresponding to a speech utterance (123);

Processing the content feed (120) to generate a semantically rich, structured document (300), the structured document (300) comprising a transcription (310) of the speech utterance (123), the transcription (310) comprising a plurality of words (312) each aligned with a corresponding audio segment (222) of the audio data (122), the corresponding audio segment indicating a time at which the word (312) was recognized in the audio data (122);

during playback of the content feed (120):

receiving a query (112) from a user requesting information contained in the content feed (120); and

Processing the query (112) and the structured document (300) by a large language model (180) to generate a response (182) to the query (112), the response (182) conveying requested information contained in the content feed (120); and

The response (182) to the query (112) is provided for output from a user device (10) associated with the user.

2. The computer-implemented method (500) of claim 1, wherein the operations further comprise:

Extracting a segment of the transcription (310) that includes the requested information conveyed by the response (182) to the query (112), the segment of the transcription (310) being defined by a start word (312) and an end word (312);

Identifying a starting audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the starting word (312) defining the segment of the transcription (310); and

Identifying an ending audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the ending word (312) defining the segment of the transcription (310),

Wherein providing the response (182) to the query (112) includes replaying the audio data (122) from the beginning audio segment of the audio data (122) to the ending audio segment of the audio data (122) from the user device (10) associated with the user.

3. The computer-implemented method (500) of claim 2, wherein:

The content feed (120) further comprises image data (124), the image data comprising a plurality of image frames (125); and

The operations further comprise: playback of the plurality of image frames (125) of the image data (124) is paused while the audio data (122) is replayed from the beginning audio segment of the audio data (122) to the ending audio segment of the audio data (122).

4. A computer-implemented method (500) as claimed in any of claims 1 to 3, wherein:

The content feed (120) further comprises image data (124), the image data comprising a plurality of image frames (125); and

The semantically rich, structured document (300) further includes author-provided text recognized in one or more image frames (125) of the plurality of image frames (125), the author-provided text aligned with a corresponding audio segment (222) of the audio data (122) to indicate a time at which the author-provided text was recognized in the one or more image frames (125).

5. The computer-implemented method (500) of claim 4, wherein processing the content feed (120) to generate the semantically rich, structured document (300) comprises: annotating the transcription (310) of the speech utterance (123) with the author-provided text by inserting the author-provided text between a pair of adjacent words (312) in the transcription (310) based on the corresponding audio segments (222) of the audio data (122) that are aligned with the author-provided text identified in the one or more image frames (125).

6. The computer-implemented method (500) of any of claims 1 to 5, wherein the response (182) to the query (112) includes a text response (182) conveying the requested information as a coherent centralized response (182) to the query (112).

7. The computer-implemented method (500) of claim 6, wherein the operations further comprise:

Performing a text-to-speech conversion on the text response (182) to generate a synthesized speech representation of the response (182) to the query (112),

Wherein providing the response (182) to the query (112) for output from the user device (10) comprises audibly outputting the synthesized speech representation of the response (182) to the query (112) from the user device (10).

8. The computer-implemented method (500) of claim 7, wherein the operations further comprise: playback of the content feed (120) is paused while the synthesized speech representation of the response (182) to the query (112) is audibly output from the user device (10).

9. The computer-implemented method (500) of any of claims 6 to 8, wherein the text response (182) to the query (112) further includes one or more references to source material related to the requested information.

10. The computer-implemented method (500) of any of claims 1 to 9, wherein the large language model (180) comprises a pre-trained large language model (180) and the structured document (300) is used as a context for the query (112) to generate the response (182) to the query (112) to perform a low sample learning.

11. The computer-implemented method (500) of any of claims 1 to 10, wherein:

the query (112) includes questions in natural language; and

The response (182) to the query (112) includes a natural language response (182) to the question.

12. The computer-implemented method (500) of any of claims 1 to 11, wherein processing the content feed (120) to generate the semantically rich, structured document (300) comprises:

-splitting the audio data (122) into a plurality of audio segments (222);

Performing speaker classification on the plurality of audio segments (222) to predict a classification result (224), the classification result comprising a corresponding speaker tag (226) assigned to each audio segment (222); and

The transcription (310) of the speech utterance (123) is indexed using the corresponding speaker tag (226) assigned to each audio segment segmented from the audio data (122).

13. A system (100) comprising:

data processing hardware (134); and

Memory hardware (136) in communication with the data processing hardware (134), the memory hardware (136) storing instructions that, when executed on the data processing hardware (134), cause the data processing hardware (134) to perform operations comprising:

Receiving a content feed (120) comprising audio data (122), the audio data (122) corresponding to a speech utterance (123);

Processing the content feed (120) to generate a semantically rich, structured document (300), the structured document (300) comprising a transcription (310) of the speech utterance (123), the transcription (310) comprising a plurality of words (312) each aligned with a corresponding audio segment (222) of the audio data (122), the corresponding audio segment indicating a time at which the word (312) was recognized in the audio data (122);

during playback of the content feed (120):

receiving a query (112) from a user requesting information contained in the content feed (120); and

Processing the query (112) and the structured document (300) by a large language model (180) to generate a response (182) to the query (112), the response (182) conveying requested information contained in the content feed (120); and

The response (182) to the query (112) is provided for output from a user device (10) associated with the user.

14. The system (100) of claim 13, wherein the operations further comprise:

Extracting a segment of the transcription (310) that includes the requested information conveyed by the response (182) to the query (112), the segment of the transcription (310) being defined by a start word (312) and an end word (312);

Identifying a starting audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the starting word (312) defining the segment of the transcription (310); and

Identifying an ending audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the ending word (312) defining the segment of the transcription (310),

Wherein providing the response (182) to the query (112) includes replaying the audio data (122) from the beginning audio segment of the audio data (122) to the ending audio segment of the audio data (122) from the user device (10) associated with the user.

15. The system (100) of claim 14, wherein:

The content feed (120) further comprises image data (124), the image data comprising a plurality of image frames (125); and

The operations further comprise: playback of the plurality of image frames (125) of the image data (124) is paused while the audio data (122) is replayed from the beginning audio segment of the audio data (122) to the ending audio segment of the audio data (122).

16. The system (100) of any one of claims 13 to 15, wherein:

The content feed (120) further comprises image data (124), the image data comprising a plurality of image frames (125); and

The semantically rich, structured document (300) further includes author-provided text recognized in one or more image frames (125) of the plurality of image frames (125), the author-provided text aligned with a corresponding audio segment (222) of the audio data (122) to indicate a time at which the author-provided text was recognized in the one or more image frames (125).

17. The system (100) of claim 16, wherein processing the content feed (120) to generate the semantically rich, structured document (300) includes: annotating the transcription (310) of the speech utterance (123) with the author-provided text by inserting the author-provided text between a pair of adjacent words (312) in the transcription (310) based on the corresponding audio segments (222) of the audio data (122) that are aligned with the author-provided text identified in the one or more image frames (125).

18. The system (100) of any of claims 13 to 17, wherein the response (182) to the query (112) includes a text response (182) conveying the requested information as a coherent centralized response (182) to the query (112).

19. The system (100) of claim 18, wherein the operations further comprise:

Performing a text-to-speech conversion on the text response (182) to generate a synthesized speech representation of the response (182) to the query (112),

Wherein providing the response (182) to the query (112) for output from the user device (10) comprises audibly outputting the synthesized speech representation of the response (182) to the query (112) from the user device (10).

20. The system (100) of claim 19, wherein the operations further comprise: playback of the content feed (120) is paused while the synthesized speech representation of the response (182) to the query (112) is audibly output from the user device (10).

21. The system (100) of any of claims 18 to 20, wherein the text response (182) to the query (112) further includes one or more references to source material related to the requested information.

22. The system (100) of any one of claims 13 to 21, wherein the large language model (180) comprises a pre-trained large language model (180) and the structured document (300) is used as a context for the query (112) to generate the response (182) to the query (112) to perform a low sample learning.

23. The system (100) of any one of claims 13 to 22, wherein:

the query (112) includes questions in natural language; and

The response (182) to the query (112) includes a natural language response (182) to the question.

24. The system (100) of any one of claims 13 to 23, wherein processing the content feed (120) to generate the semantically rich, structured document (300) comprises:

-splitting the audio data (122) into a plurality of audio segments (222);

Performing speaker classification on the plurality of audio segments (222) to predict a classification result (224), the classification result including a corresponding speaker tag (226) assigned to each audio segment; and

The transcription (310) of the speech utterance (123) is indexed using the corresponding speaker tag (226) assigned to each audio segment segmented from the audio data (122).