DE102022105123A1 - Interact with semantic video segments through interactive tiles - Google Patents

Abstract

Embodiments relate to interactive tiles representing video segments of a segmentation of video. In some embodiments, each interactive tile represents a different video segment from a particular video segmentation (e.g., standard video segmentation). Each interactive tile includes a preview image (e.g., the first frame of the video segment represented by the tile), a transcript from the beginning of the video segment , a visualization of detected faces in the video segment, and one or more faceted timelines visualizing a category of detected features (e.g., visualization of detected visual scenes, audio classifications, visual artifacts). In some embodiments, interacting with a particular interactive tile causes navigating to a corresponding portion of the video, adding a corresponding video segment to a selection, and/or swiping through tile thumbnails.

Description

Reference to related application

This application is a continuation-in-part of U.S. patent application no. 17/017,344 with the invention title “Segmentation and Hierarchical Clustering of Video”; the entire content of which is hereby incorporated by reference.

background

In recent years, the use of videos has become very widespread; Videos have applications in almost every industry from film and television to advertising and social media. Companies and individuals routinely create and share video content in a variety of contexts, such as presentations, tutorials, commentary, news and sports segments, blogs, product reviews, testimonials, comedy, dance, music, movies, and video games, to name just a few to call. Video can be captured using a camera, generated using animation or rendering tools, edited using different types of video editing software, and shared through a variety of output channels. Recent advancements in digital cameras, smartphones, social media, and other technologies have provided a number of new ways that make it easier for even non-professionals to capture and share video. However, with these new ways of producing and sharing videos, the need for video editing features increases.

Traditionally, video editing involves selecting video frames and performing some type of action on the frames or the associated sound. Some common operations include importing, trimming, cropping, reordering, applying transitions and effects, adjusting colors, adding captions and graphics, exporting, and others. Video editing software, such as ADOBE® PREMIERE® PRO and ADOBE PREMIERE ELEMENTS, typically includes a graphical user interface (GUI) that presents a video timeline that represents the video frames in the video and allows the user to select specific frames and those attached to the selects operations to be performed on frames. However, traditional video editing can be cumbersome, complicated and beyond the capabilities of many users.

summary

Embodiments of the present invention relate to video segmentation and various interaction modalities for video browsing, editing and playing. In an exemplary embodiment, a video is incorporated by detecting various features using one or more machine learning models and generating one or more video segmentations based on the detected features. The detected features serve as the basis for one or more video segmentations, such as a standard segmentation, a search segmentation based on a user query, a snap point segmentation that identifies selection snap points, and/or a thumbnail segmentation that identifies which parts of a video timeline are to be displayed with different thumbnail images . Finder and editor interfaces present the various segmentations and allow users to browse, select, edit, and/or play semantically meaningful video segments.

In some embodiments, a segmentation of a video is computed by determining candidate boundaries from detected feature boundaries from one or more feature lanes; modeling different segmentation options by building a graph with nodes representing candidate boundaries, edges representing candidate segments, and edge weights representing cut costs; and computing the video segmentation by solving a shortest path problem to find the path through the edges (segmentation) that minimizes the sum of edge weights along the path (cutting cost). In some embodiments, one or more aspects of a segmentation routine depend on the type of segmentation (e.g., standard, search, snap point, thumbnail). As a non-limiting example, candidate cut points are determined differently, different edges are used for different types of segmentations, and/or different cut costs are used for edge weights. In an exemplary embodiment, a default segmentation is selected from a desired th set of detected features, such as from detected sentences, faces and visual scenes.

In some embodiments, a viewfinder interface presents a video segmentation, such as a standard segmentation, with interactive tiles representing the video segments in the video segmentation and detected features in each video segment. In some embodiments, each interactive tile represents a different video segment from a particular video segmentation (e.g., a standard video segmentation) and includes a preview image (e.g., the first frame of the video segment represented by the tile), a transcript from the beginning of the video segment, a visualization of detected faces in the video segment and/or one or more faceted timelines visualizing a category of detected features (eg, a visualization of detected visual scenes, audio classifications, visual artifacts). In one embodiment, various ways of interacting with a particular interactive tile are for navigating to a corresponding portion of the video, adding a corresponding video segment to a selection, and/or swiping through tile thumbnails.

In some embodiments, a search segmentation is calculated based on a query. First, a first segmentation, such as a default segmentation, is displayed (e.g., as interactive tiles in a finder interface, as a video timeline in an editor interface), and the default segmentation is re-segmented in response to a user request. The query may take the form of a keyword and one or more selected facets in a category of detected features. Keywords in detected transcript words, detected object or action tags, or detected audio event tags that match the keywords are searched for. Selected facets are searched for in detected instances of the selected facets. Each video segment that matches the query is re-segmented by solving a shortest-path problem through a graph that models various segmentation options. The searcher interface updates the interactive tiles to reflect the search segmentation. Browsing is therefore for breaking the interactive tiles to provide smaller chunks of video based on a request.

In some embodiments, the viewfinder interface is used to browse video and add video segments to a selection. Upon adding a desired set of video segments to a selection, a user switches to an editor interface to perform one or more refinements or other editing operations on the selected video segments. In some embodiments, the editor interface initializes a video timeline with a representation of the video segments selected from the viewfinder interface, such as a composite video timeline that represents a composite video formed by the selected video segments, with boundaries of the video segments that are shown as an underlay to the composite timeline. In some embodiments, the composite video timeline includes a visualization of detected features and corresponding feature areas to aid in selecting, trimming, and editing video segments.

Some embodiments relate to snap point segmentation that defines the locations of selection snap points for selecting video segments through interactions with the video timeline. Candidate snap points are determined from boundaries of feature regions of the video that indicate when instances of detected features are present in the video. In some embodiments, candidate snap point separations are penalized if they are separated by less than a minimum duration corresponding to a minimum pixel separation between consecutive snap points on a video timeline. The snap point segmentation is computed by solving a shortest path problem through a graph that models different snap point locations and separations. When a user clicks or taps on the video timeline and drags, a selection snaps to the snap points defined by the snap point segmentation. In some embodiments, the snap points appear during a drag operation and disappear when the drag operation ends.

Some embodiments relate to thumbnail segmentation that defines the locations on a video timeline where thumbnails are displayed. Candidate preview image locations are determined from feature region boundaries of the video, indicating when instances of detected features are present in the video. In some embodiments, candidates Thumbnail separations are penalized if they are separated by less than a minimum duration equal to a minimum pixel separation (e.g., the width of a thumbnail) between consecutive thumbnail locations on a video timeline. Thumbnail segmentation is computed by solving a shortest path problem through a graph that models various thumbnail locations and separations. As such, a video timeline is displayed with thumbnails on the timeline at locations defined by the thumbnail segmentation, each thumbnail representing a portion of the video associated with the thumbnail location.

In some embodiments, the editor interface provides any number of editing functions for a selected video segment. Depending on the implementation, available editing features include stylistic refinements that transform content (e.g., reducing wind noise), duration-affecting refinements that eliminate content (e.g., "hiding" areas of footage, removing profanity, realizing a time warp, truncation to n seconds), and/or context functions that depend on the selected context (e.g. remove or beep words from the content with an appropriate transcript). In general, the editor interface provides any suitable editing functionality, including rearranging, cropping, applying transitions or effects, adjusting colors, adding captions or graphics, and/or others. The resulting composite video can be played, saved, exported, or otherwise.

As such, the present disclosure provides intuitive video interaction techniques that allow users to easily select, trim, play, and export semantically meaningful video segments at a desired level of granularity, providing creators and consumers with a more intuitive structure for interacting with videos is handed over.

This summary is provided to introduce in simplified form a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

character list

• 1A und 1B sind Blockdiagramme eines exemplarischen Rechensystems zum Bearbeiten oder Abspielen von Videos entsprechend Ausführungsformen der vorliegenden Erfindung.
• 2A bis 2F sind Darstellungen einer exemplarischen Technik zum Berechnen von Kandidatengrenzen aus detektierten Merkmalsgrenzen entsprechend Ausführungsformen der vorliegenden Erfindung.
• 3A bis 3C sind Darstellungen einer exemplarischen Technik zum Aufbauen eines Graphen mit Wegen, die verschiedene Kandidatensegmentierungen darstellen, entsprechend Ausführungsformen der vorliegenden Erfindung.
• 4A bis 4C sind Darstellungen einer exemplarischen Technik zum Berechnen von Grenzschneidekosten auf Grundlage von visuellen Szenen entsprechend Ausführungsformen der vorliegenden Erfindung.
• 5 ist eine Darstellung einer exemplarischen Technik zum Berechnen von Grenzschneidekosten auf Grundlage von diarisierten (diarized) Transkriptmerkmalen entsprechend Ausführungsformen der vorliegenden Erfindung.
• 6 ist eine Darstellung einer exemplarischen Technik zum Berechnen von Grenzschneidekosten auf Grundlage von detektierten Gesichtern entsprechend Ausführungsformen der vorliegenden Erfindung.
• 7A bis 7D sind Darstellungen einer exemplarischen Technik zum Berechnen von Grenzschneidekosten auf Grundlage einer Anfrage entsprechend Ausführungsformen der vorliegenden Erfindung.
• 8 ist eine Darstellung einer exemplarischen Technik zum Berechnen von Intervallschneidekosten auf Grundlage einer Inkohärenz von detektierten Merkmalen entsprechend Ausführungsformen der vorliegenden Erfindung.
• 9A und 9B sind Darstellungen einer exemplarischen Technik zum Berechnen von Intervallschneidekosten auf Grundlage einer Anfrage entsprechend Ausführungsformen der vorliegenden Erfindung.
• 10 ist eine Darstellung einer exemplarischen Sucherschnittstelle zum Durchlaufen bzw. Browsen von Standard- und/oder Suchsegmentierungen entsprechend Ausführungsformen der vorliegenden Erfindung.
• 11A und 11B sind Darstellungen von exemplarischen interaktiven Kacheln entsprechend Ausführungsformen der vorliegenden Erfindung.
• 12A bis 12D sind Darstellungen von exemplarischen facettenbestückten Suchmenüs entsprechend Ausführungsformen der vorliegenden Erfindung.
• 13 ist eine Darstellung einer exemplarischen Suche in einer Sucherschnittstelle entsprechend Ausführungsformen der vorliegenden Erfindung.
• 14 ist eine Darstellung einer exemplarischen Bearbeiterschnittstelle zur Videobearbeitung entsprechend Ausführungsformen der vorliegenden Erfindung.
• 15A und 15B sind Darstellungen einer exemplarischen Marquee-Auswahl mit Einrastung entsprechend Ausführungsformen der vorliegenden Erfindung.
• 16A und 16B sind Flussdiagramme zur Darstellung eines Verfahrens zum Generieren einer Videosegmentierung unter Nutzung eines graphischen Modells entsprechend Ausführungsformen der vorliegenden Erfindung.
• 17 ist ein Flussdiagramm zur Darstellung eines Verfahrens zum Segmentieren einer Standardsegmentierung eines Videos zu einer Suchsegmentierung entsprechend Ausführungsformen der vorliegenden Erfindung.
• 18 ist ein Flussdiagramm zur Darstellung eines Verfahrens zum Navigieren eines Videos unter Nutzung von interaktiven Kacheln entsprechend Ausführungsformen der vorliegenden Erfindung.
• 19 ist ein Flussdiagramm zur Darstellung eines Verfahrens zum Hinzufügen von Videosegmenten zu einer Auswahl entsprechend Ausführungsformen der vorliegenden Erfindung.
• 20 ist ein Flussdiagramm zur Darstellung eines Verfahrens zum Einrasten einer Auswahlgrenze einer Auswahl von Videosegmenten entsprechend Ausführungsformen der vorliegenden Erfindung.
• 21 ist ein Flussdiagramm zur Darstellung eines Verfahrens zum Präsentieren eines Videozeitstrahles mit Vorschaubildern an Orten, die durch eine Vorschaubildsegmentierung definiert sind, entsprechend Ausführungsformen der vorliegenden Erfindung.
• 22 ist ein Flussdiagramm zur Darstellung eines weiteren Verfahrens zum Präsentieren eines Videozeitstrahles mit Vorschaubildern an Orten, die durch eine Vorschaubildsegmentierung definiert sind, entsprechend Ausführungsformen der vorliegenden Erfindung.
• 23 ist ein Blockdiagramm einer exemplarischen Rechenumgebung, die zur Nutzung bei der Implementierung von Ausführungsformen der vorliegenden Erfindung geeignet ist.

The present invention is described in detail below with reference to the attached drawing figures.

• 1A and 1B 12 are block diagrams of an exemplary computing system for editing or playing video, in accordance with embodiments of the present invention.
• 2A until 2F 12 are illustrations of an exemplary technique for computing candidate boundaries from detected feature boundaries, in accordance with embodiments of the present invention.
• 3A until 3C 10 are illustrations of an exemplary technique for constructing a graph with paths representing different candidate segmentations, according to embodiments of the present invention.
• 4A until 4C 12 are illustrations of an exemplary technique for computing a boundary cutting cost based on visual scenes, in accordance with embodiments of the present invention.
• 5 Figure 12 is an illustration of an exemplary technique for computing a boundary pruning cost based on diarized transcript features, in accordance with embodiments of the present invention.
• 6 12 is an illustration of an exemplary technique for calculating a boundary clipping cost based on detected faces, in accordance with embodiments of the present invention.
• 7A until 7D 12 are illustrations of an exemplary technique for calculating a boundary cutting cost based on a query, in accordance with embodiments of the present invention.
• 8th 12 is an illustration of an exemplary technique for calculating an interval cutting cost based on incoherence of detected features, in accordance with embodiments of the present invention.
• 9A and 9B 12 are illustrations of an exemplary technique for calculating an interval cutting cost based on a query, in accordance with embodiments of the present invention.
• 10 Figure 12 is an illustration of an example searcher interface for browsing standard and/or search segmentations, in accordance with embodiments of the present invention.
• 11A and 11B 12 are representations of exemplary interactive tiles in accordance with embodiments of the present invention.
• 12A until 12D 10 are representations of exemplary faceted search menus according to embodiments of the present invention.
• 13 Figure 12 is an illustration of an exemplary search in a searcher interface, in accordance with embodiments of the present invention.
• 14 Figure 12 is an illustration of an example editor interface for video editing, in accordance with embodiments of the present invention.
• 15A and 15B 12 are illustrations of an exemplary marquee selection with snapping in accordance with embodiments of the present invention.
• 16A and 16B 12 are flow charts illustrating a method for generating video segmentation using a graphical model according to embodiments of the present invention.
• 17 Fig. 12 is a flow chart depicting a method for segmenting a standard segmentation of a video into a search segmentation in accordance with embodiments of the present invention.
• 18 FIG. 12 is a flow chart depicting a method for navigating a video using interactive tiles in accordance with embodiments of the present invention.
• 19 Fig. 12 is a flow chart depicting a method for adding video segments to a selection according to embodiments of the present invention.
• 20 Fig. 12 is a flow chart depicting a method for snapping a selection boundary to a selection of video segments in accordance with embodiments of the present invention.
• 21 Fig. 12 is a flow diagram illustrating a method for presenting a video timeline with thumbnails at locations defined by thumbnail segmentation, according to embodiments of the present invention.
• 22 Figure 12 is a flow diagram illustrating another method for presenting a video timeline with thumbnails at locations defined by thumbnail segmentation, according to embodiments of the present invention.
• 23 Figure 1 is a block diagram of an example computing environment suitable for use in implementing embodiments of the present invention.

detailed description

overview

Typically, a video file, clip, or project can be divided into visual and audio elements. For example, a video may encode or otherwise identify a video track that includes a sequence of still images (e.g., video frames) and an accompanying audio track that includes one or more audio signals. Typically, video editing tools provide an interface that allows users to perform time-based editing on selected video frames. Conventional video editing therefore generally involves presenting a video as a succession of fixed units of equal duration (eg, video frames) and presenting a video timeline that allows the user to select and interact with specific video frames. However, the interaction modalities based on a selection of specific video frames or a corresponding time range are inherently slow and detailed, resulting in editing workflows that are often considered cumbersome, complicated, or beyond the capabilities of many users. In other words, a time-based video editing that involves selecting of specific video frames or time ranges provides a fixed granularity interaction modality, resulting in an inflexible and inefficient interface. As such, there is a need for an improved interface and interaction modalities for video editing tools.

Accordingly, embodiments of the present invention relate to video segmentation and various interaction modalities for video browsing, editing, and playback. In an exemplary embodiment, a video is ingested by detecting various features using one or more machine learning models, generating one or more video segmentations based on the detected features. Specifically, the video is processed using one or more machine learning models to identify features (e.g., a transcript, linguistic features, speakers, faces, audio classifications, visually similar scenes, visual artifacts, video objects or actions, audio events, software log events) and corresponding feature regions , in which the detected features are present. The detected features serve as the basis for one or more video segmentations, such as standard segmentation, search segmentation based on a user query, snap point segmentation identifying selection snap points, and/or thumbnail segmentation identifying which parts of a video timeline are presented with different thumbnail images must. Finder and editor interfaces present the various segmentations while providing users with the ability to browse, select, edit, and/or play semantically meaningful video segments. As such, the present techniques provide new ways to create, edit, and consume videos, thereby providing creators and consumers with a more intuitive structure for interacting with videos.

In an exemplary implementation, video segmentation is identified by determining candidate intersection points from detected feature boundaries; modeling various segmentation options by building a graph with nodes representing candidate intersection points, edges representing candidate segments, and edge weights representing intersection costs; and computing the video segmentation by solving a shortest path problem to find the path through the edges (segmentation) that minimizes the sum of edge weights along the path (slicing cost). In some embodiments, a segmentation routine accepts various input parameters depending on the use case, such as designated feature tracks (e.g., predetermined, user-selected and/or other), user requests, a minimum or maximum target length of a video segment (which in some cases depends on the zoom level), an area of a video for segmentation and/or other. Additionally or alternatively, one or more aspects of a segmentation routine depend on the type of segmentation. As a non-limiting example, candidate cut points are determined differently, different edges are used for different types of segmentations, and/or different cut costs are used for edge weights. In some embodiments, the output is a representation of a complete set (i.e., covering the entire and/or designated area of the video) of disjoint (i.e., non-overlapping) video segments.

In an exemplary embodiment, a standard segmentation is computed from a desired set of detected features, such as detected sentences, faces, and visual scenes. A viewfinder interface presents the default segmentation with interactive tiles showing the video segments in the default segmentation and detected features in each video segment. In some embodiments, the searcher interface presents the categories of features used by the default segmentation, accepts a selection of a desired set of feature categories, recalculates the default segmentation based on the selected feature categories, and updates the interactive tiles to reflect the updated default segmentation. In an exemplary implementation with a search, the searcher interface accepts a query (such as a keyword and/or facet), generates or triggers search segmentation that resegments the default segmentation based on the query, and updates the interactive tiles to reflect the search segmentation to represent. Therefore, searching is for breaking the interactive tiles to display smaller chunks of video based on the query. Additionally or alternatively, a slider or other interaction element presents an input parameter for segmentation (e.g., a minimum and maximum target length of the video segment), allowing a user to interactively control the size of video segments represented by the tiles .

In an exemplary embodiment, each interactive tile represents a video segment from a particular segmentation (e.g. standard or search) and detected features in the video segment. Each tile shows, for example, a preview image (e.g. the first frame of the video segment), a transcript from the beginning of the video segment , a visualization of detected faces in the video segment, and one or more faceted timelines visualizing a category of detected features (e.g., visualization of detected visual scenes, audio classifications, visual artifacts). Clicking on a visualization of a detected feature in an interactive tile causes navigation to that part of the video with the detected feature.

In some embodiments, the viewfinder interface includes a selected clips panel where users can add video segments to a selection. Depending on the implementation, video segments are added to the selection in various ways, such as dragging a tile into the box intended for selected clips, activating a button or other interaction element in an interactive tile, interacting with a visualization of a detected feature in an interactive tile (e.g. right-clicking on a visualization to activate a context menu and adding a corresponding video segment to select from the context menu), interacting with the transcript (e.g. highlighting, right-clicking to activate a context menu and adding to selection from the context menu) and/or in other ways. In some embodiments, the viewfinder interface includes one or more buttons or other interaction elements that toggle to the editor interface, thereby initializing a video timeline with a representation of the selected video segments in the selected clips field.

In an exemplary embodiment, the editor interface provides a composite video timeline that represents a composite video formed from the video segments selected in the viewfinder interface, with boundaries of the video segments appearing as an underlay to the composite timeline being represented. In some embodiments, dragging along the composite timeline causes selection boundaries to snap to snap points defined by snap point segmentation and/or a current zoom level. In an exemplary implementation with a search, the agent interface accepts a query (e.g., a keyword and/or a facet), generates a search segmentation that segments the video segments in the composite video based on the search, and presents a visualization of the search segmentation (e.g., by displaying the boundaries of the video segments as an underlay to the composite timeline).

In some embodiments, the composite video timeline includes a visualization of detected features and corresponding feature areas to aid in trimming. For example, while dragging along the composite timeline, visualizations of detected features can help inform which parts of the video to select (e.g., a part of the video that contains a visualized audio classification). Additionally or alternatively, while dragging along the composite timeline, the snap points defined by the snap point segmentation and representing certain feature boundaries are visualized on the timeline (e.g., as vertical lines on the composite timeline), representing which parts of the video are good snap points. In yet another example, clicking on a visualization of a detected feature (e.g., a bar representing a portion of the video with a detected artifact) causes selection of that portion of the composite video with the detected feature. As such, the editor interface provides an intuitive interface for selecting semantically meaningful video segments for editing.

In some embodiments, the composite timeline represents each video segment in the composite video with one or more thumbnail images that show a representative video frame. In an exemplary implementation, each video segment includes one or more thumbnail images at locations of the timeline determined by the thumbnail segmentation and/or a current zoom level can be identified, with longer video segments more likely to include multiple thumbnails. For example, if a user zooms in, more thumbnails appear in semantically meaningful locations, while any thumbnails that are already visible remain in place. As such, they serve Thumbnails as landmarks to aid in navigating through a video and selecting video segments.

In an exemplary implementation, the editor interface provides any number of editing functions for a selected video segment. Depending on the implementation, available editing features include stylistic refinements that transform content (e.g., reduce wind noise), duration-affecting refinements that eliminate content (e.g., "hide" areas of footage, remove swear words, perform a time warp, truncation to n seconds make), and/or contextual functions that depend on the selected content (which, for example, remove or beep words from the content with a corresponding transcript). In general, the editor interface provides any suitable editing functionality, including rearranging, cropping, applying transitions or effects, adjusting colors, adding captions or graphics, and/or others. The resulting composite video can be played, saved, exported, or otherwise.

As such, the present disclosure provides intuitive video interaction techniques that allow users to easily select, trim, play, and export semantically meaningful video segments at a desired level of granularity. Instead of merely providing a video timeline segmented by arbitrary fixed units of equal duration (e.g., one frame, one second) in a way that is independent of semantic meaning, presenting video segmentation based on detected features provides video segments of unequal duration and borders that are in semantically meaningful and not arbitrary places. Accordingly, the present video interaction techniques provide a more flexible and efficient interaction modality that allows users to quickly identify, select, and edit meaningful chunks of a video that are likely to be of interest. As such, editors can now work faster and consumers can now jump to a selection of interest without having to watch the video.

Exemplary video editing environment

1A 10 shows a block diagram of an exemplary environment 100 suitable for use in implementing embodiments of the present invention. In general, the environment 100 is suitable for editing or playing videos and facilitates, among other things, video segmentation and interactions with resulting video segments. The environment 100 includes a client device 102 and a server 150. In various embodiments, the client device 102 and/or the server 150 are any type of computing device that can facilitate editing or playing of videos, such as the computing device 2300 described below with reference to 23 will be described later. Examples of computing devices include a personal computer, laptop computer, mobile or mobile device, smartphone, tablet computer, smart watch, wearable computer, personal digital assistant (PDA), music player, or MP3 player. a global positioning system (GPS) or device, a video player, a handheld communication device, a gaming device or system, an entertainment system, a vehicle computing system, an embedded system controller, a camera, a remote control, a barcode scanner, a computerized measuring device, an appliance , a consumer electronic device, a workstation, a combination thereof, or any other suitable computing device.

In various implementations, the components of the environment 100 include computer storage media for storing information, including data, data structures, computer instructions (e.g., software program instructions, routines, or services), and/or models (e.g., machine learning models) used in some embodiments of the technologies described herein will. For example, in some implementations, client device 102, server 150, and/or memory 190 comprise one or more data stores (or computer memory). Although client device 102, server 150, and storage 190 in 1A Also, while each is illustrated as a single component, in some embodiments client device 102, server 150, and/or storage 190 are implemented using any number of data stores and/or are implemented using cloud storage.

The components of environment 100 communicate with one another over a network 103. In some embodiments, network 103 includes one or more local area networks (LANs), Wide Area Networks (WANs) and/or other networks. Such networked environments are common in offices, enterprise computer networks, intranets, and the Internet.

At the in 1A and 1B In the example illustrated, client device 102 includes video interaction engine 108, and server 150 includes video segmentation tool 155. In various embodiments, video interaction engine 108, video segmentation tool 155, and/or any of 1A and 1B elements shown are installed or integrated into an application/applications (for example a corresponding application on the client device 102 or the server 150) or an add-on/add-ons or a plug-in/plug-ins for an application/applications. In some embodiments, the application(s) is/are any application that can facilitate video editing or playing, such as a standalone application, a mobile application, a web application, and/or the like. In some implementations, the application(s) comprises a web application accessible, for example, via a web browser, hosted at least in part server-side, and/or the like. Additionally or alternatively, the application/applications include/have a specially provided or dedicated application. In some cases, the application (e.g., as a service) is integrated into an operating system. Exemplary video editing applications include ADOBE PREMIERE PRO and ADOBE PREMIERE ELEMENTS.

In various embodiments, the functionality described herein is distributed across any number of devices. In some embodiments, the video editing application 105 is hosted, at least in part, server-side such that the video interaction engine 108 and the video segmentation tool 155 work together (e.g., over the network 103) to perform the functionality described herein. In another example, the video interaction engine 108 and the video segmentation tool 155 (or any portion thereof) are integrated into a common application executable on a single device. Although some embodiments are described in terms of an application(s), any of the functionality described herein is additionally or alternatively implemented in an operating system (e.g., as a service), a server (e.g., a remote server), a distributed computing environment (e.g., as a cloud service), and/or other integrated. However, these are merely examples and any suitable distribution of functionality between these or other devices may be implemented within the scope of the present disclosure.

Starting with a high-level overview of an example workflow through the configuration included in 1A and 1B is shown. Client device 102 is a desktop, laptop, or mobile device, such as a tablet or smartphone. The video editing application 105 provides one or more user interfaces. In some embodiments, a user accesses a video through video editing application 105 and/or otherwise uses video editing application 105 to identify the location where a video is stored (whether locally on client device 102, at a remote location, such as in memory 190, or elsewhere). Additionally or alternatively, a user records video using video recording capabilities of the client device 102 (or any other device) and/or any application running at least in part on the device (e.g., ADOBE BEHANCE). In some cases, the video editing application 105 uploads the video (e.g., to an accessible storage 190 for video files 192) or otherwise communicates the location of the video to the server 150, where the video segmentation tool 155 receives or accesses the video and one or performs multiple onboarding functions on the video.

In some embodiments, the video segmentation tool 155 extracts various features from the video (e.g., a transcript, linguistic features, speakers, faces, audio classifications, visually similar scenes, visual artifacts, objects or actions from videos, audio events, software log events) and generates and stores a representation of the detected features, the corresponding feature regions in which the detected features are present, and/or the corresponding confidence levels (e.g., detected features 194). In an exemplary implementation, the video segmentation tool 155 generates and stores, based on the detected features, a representation of one or more segmentations of the video (e.g., a video segmentation(s) 196), such as a default segmentation, a search segmentation based on a user query, a snap-point segmentation, and/or a Thumbnail segmentation. In some cases, one or more of the segmentations are set to different which generates levels of granularity (e.g. corresponding to different zoom levels). In some embodiments, some segmentation is generated at ingestion. In some cases, some or all of the segmentation is generated at a different time (for example, upon request). As such, the video segmentation tool 155 and/or video editing application 105 accesses a video (e.g., one of the video files 192) and generates and stores a representation of one or more segmentations of the video (e.g., the video segmentation(s) 196), of constituent semantically meaningful video segments the video segmentation(s) (e.g., video files 192) and/or any representation thereof in any suitable storage location, such as in memory 190, on client device 102, on server 150, on a combination thereof, and/or other locations.

In an exemplary embodiment, the video editing application 105 (e.g., the video interaction engine 108) provides one or more user interfaces with one or more interaction elements that enable a user to interact with the embedded video, and more specifically with one or more video segmentations 196 (e.g., with semantically meaningful video segments , snap points, thumbnail locations) and/or the detected features 194 interacts. 1 B 10 shows an exemplary implementation of the video interaction engine 108 that includes a video walkthrough tool 110 and a video editing tool 130. FIG.

In an exemplary implementation, the video walkthrough tool 110 provides a viewfinder interface that presents a standard segmentation with interactive tiles 112 that represent the video segments in the standard segmentation and the detected features 194 in each video segment. In an exemplary implementation with a search, the video walkthrough tool 110 includes a search re-segmentation tool 118 that accepts a query (e.g., a keyword and/or facet), a search segmentation (e.g., generated by the search segmentation component 170 of FIG 1A generated) that resegments, generates, or otherwise triggers the default segmentation based on the query, and updates the interactive tiles 112 to reflect the search segmentation. The video walkthrough tool 110 provides a selected clips pane 114 where users can add video segments to a selection.

The video editing tool 130 provides an editor interface with a composite clip timeline tool 132 presenting a composite video timeline representing composite video made up of the video segments selected in the viewfinder interface. In some embodiments, a marquee selection and snapping tool 106 detects a drag along the composite timeline and snaps selection boundaries to snap points defined by the snap point segmentation and/or a current zoom level. In some cases, the tool 132 intended for the composite clip timeline includes a feature visualization tool 134 that presents a visualization of detected features 194 and corresponding feature regions to aid in trimming. Additionally or alternatively, the thumbnail preview tool 138 displays one or more thumbnails showing a representative video frame on the composite timeline at locations identified by the thumbnail segmentation and/or a current zoom level. As such, the video editing tool 130 allows users to navigate through a video and select semantically meaningful video segments.

Depending on the implementation, the video editing tool 130 and/or the video interaction engine 108 perform any number and variety of operations on selected video segments. As a non-limiting example, selected video segments are played, deleted, otherwise edited (e.g., reordering, cropping, applying transitions or effects, adjusting color, adding captions or graphics), exported, and/or modified by other operations. Therefore, in various embodiments, the video interaction engine 108 provides interface functionality that allows a user to select, navigate, play, and/or edit a video based on interactions with semantically meaningful video segments and detected features thereof.

Exemplary video segmentation techniques

As in 1A As shown, in some embodiments, the video segmentation tool 155 generates a representation of one or more segmentations of a video. Generally, one or more segmentations are generated at any suitable time, such as at ingesting or initially processing a video (e.g., in standard segmentation), upon receiving a request (e.g., in search segmentation), displaying a video timeline, activating a user interface, and/or at any other time.

At the in 1A In the example shown, the video incorporation tool 160 incorporates a video (e.g., a video file, a portion of a video file, a video represented or otherwise identified by a project file). In some embodiments, incorporating a video includes extracting one or more features from the video and/or generating one or more segmentations of the video to identify corresponding semantically meaningful video segments and/or segment boundaries. At the in 1A In the illustrated implementation, the video incorporation tool 160 includes a feature extraction component/components 162 and a standard segmentation component 164. In this implementation, the feature extraction component/components detect/detect one or more features from the video, and the standard segmentation component 164 triggers a video segmentation component 180 to perform standard video segmentation during video incorporation to generate.

Considered at a high level, the video incorporation tool 160 (e.g., the feature extraction component(s) 162) involves detecting, extracting, or otherwise determining various features (e.g., a transcript, linguistic features, speakers, faces, audio classifications, visually similar scenes, visual video artifacts, objects or actions, audio events, software log events) from a video using, for example, one or more machine learning models, natural language processing, digital signal processing, and/or other techniques. In some embodiments, feature extraction component(s) 162 includes one or more machine learning models for each of multiple categories of feature for detection. As such, the video incorporation tool 160 and/or the corresponding feature extraction component(s) 162 extract, generate and/or store a representation of detected features (e.g., facets) in each category, corresponding feature regions where the detected features are present, and/or of appropriate levels of reliability.

In some embodiments, one or more feature categories (e.g., speakers, faces, audio classifications, visually similar scenes, and the like) have their own feature track that represents the instances of detected features (e.g., facets, such as unique faces or speakers) in a feature category. By way of non-limiting example, the representation of detected features (e.g., detected features 194) for each feature category includes a list, array, or other representation of each instance of a detected facet (e.g., each unique face) in the feature category (e.g. detected faces). In an example implementation, each instance of a detected facet is represented with a feature area in which the instance is detected (e.g., start and stop timestamps per instance) with a unique value identifying the facet to which the instance belongs ( e.g., with a unique value for each unique face, speaker, visual scene, and the like), with a corresponding confidence level that quantifies the reliability or likelihood of the prediction, and/or with a representation of other properties.

In some embodiments, feature extraction component(s) 162 extracts a transcript and/or linguistic features from an audio track associated with video. In an exemplary implementation, any known speech-to-text algorithm is applied to the audio track to generate a speech transcript, detect segments of speech (e.g., corresponding to words, phrases, utterances in continuous speech separated by audio gaps, and the like). to detect non-speech segments (e.g., pauses, silence, or sound without speech), and/or the like. In some embodiments, voice activity detection (e.g., on the audio track, on detected non-speech segments) is applied to detect and/or categorize segments of the audio track with human sounds other than words (e.g., laughter, audible throat clearing, and the like). In some cases, the transcript and/or detected transcript segments are associated with the video timeline, and transcript segments are associated with corresponding time ranges. In some embodiments, any known topical segmentation technique (semantic analysis, Natural language processing, application of a language model) is used to screen out or otherwise identify portions of the video that are likely to contain similar themes, assigning detected speech segments to a score representing how likely it is that the language segment is the end of a topical or topic-related segment.

In some embodiments, the feature extraction component(s) includes one or more machine learning models that detect unique speakers from an audio track associated with a video. In an exemplary implementation, any known speech recognition, speaker identification, or speaker diarization technique is applied to detect unique speech footprints (e.g., within a single video, across a collection of videos) and to screen out or otherwise assign portions of the audio track/audio tracks based on speaker identity identify. Exemplary techniques used in speech recognition, speaker identification, or speaker diarization utilize frequency estimation, pattern matching, vector quantification, decision trees, hidden Markov models, Gaussian mixture models, neural networks, and/or other techniques. In some embodiments, the feature extraction component(s) 162 apply a speaker diarization technique such as described in: "Active Speakers in Context" by Juan Leon Alcazar, Fabian Caba, Long Mai, Federico Perazzi, Joon-Young Lee, Pablo Arbelaez, Bernard Ganem; published at Proceedings of the IEEE/CVF, Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pages 12465 to 12474. In addition or as an alternative to using audio signatures to detect speakers, one or more machine learning models are used in some embodiments, to determine which detected face is speaking by detecting mouth movement on detected faces. In an exemplary implementation, each instance of a detected speaker in a video is associated with a corresponding time range of the video in which the speaker was detected, a corresponding confidence level that quantifies the reliability or likelihood of the prediction, and/or a thumbnail of the detected face of the detected speaker. Additionally or alternatively, detected speech segments (e.g., words, phrases, sentences) and/or other transcript features are associated with a representation of a corresponding detected speaker to generate a diarized transcript.

In some embodiments, feature extraction component(s) 162 includes one or more machine learning models that detect unique faces from video frames of a video. In an exemplary implementation, any known face detection technique (e.g., RetinaFace) is applied to detect unique faces in each video frame and/or over time. Each video frame is processed, for example, by segmenting each face from the background (e.g. using one or more neural networks), aligning each face, detecting locations of facial landmarks (e.g. eyes, nose, mouth) and generating a (e.g. vectorial) representation of the detected facial landmarks. In some embodiments, detected faces from different frames (e.g., within a single video, across a collection of videos) that have similar representations (e.g., separated by a distance within a threshold, are clustered based on one or more clustering algorithms) identified as belonging to the same identity. In an exemplary implementation, each instance of a detected face is associated with a corresponding time range spanning the video frames in which the face was detected and/or a corresponding confidence level that quantifies the reliability or likelihood of the prediction.

In some embodiments, feature extraction component(s) 162 includes one or more machine learning models that extract audio classifications from an audio track associated with video. Any known tone detection technique is used to detect any number of audio classes (e.g. music, speech, other). In an exemplary implementation, each frame of audio data from the audio track is encoded (e.g., using linear predictive coding) into a vector representation and classified by one or more neural networks. In some embodiments, the audio timeline is categorized into the detected audio classes (eg, music, speech, or other). In an exemplary implementation, consecutive audio frames with the same classification are grouped together and assigned a corresponding time range spanning the audio frames and/or a corresponding (e.g., averaged) confidence level that quantifies the reliability or likelihood of the prediction.

In some embodiments, feature extraction component(s) 162 includes one or more machine learning models that detect visually similar scenes from video frames of a video. In an exemplary implementation, each video frame is processed (e.g., by one or more neural networks) to extract a corresponding (e.g., vectorial) representation of visual features in the video frame, and the representations of the various video frames are combined over time into a ( e.g. fixed or variable) number of visual scenes using any suitable clustering algorithm (e.g. k-means clustering). In some embodiments, each visual scene is associated with a corresponding time range spanning the video frames in the visual scene. In some cases, each scene transition is assigned a transition reliability, for example by computing a distance metric that compares representations of the visual features in the video frames surrounding the transition.

In some embodiments, the feature extraction component(s) 162 includes one or more machine learning models that detect visual artifacts in the video frames of a video. Any known visual detection technique is used to identify one or more classes of visual artifacts from the video frames. In an exemplary implementation, one or more neural network classifiers detect a corresponding class of visual artifact, such as erroneous camera motion across video frames, camera occlusion in a given video frame, blurring in a given video frame, compression artifacts in a given video frame, the lack of motion (e.g., blank video frame, no visual changes across video frames) and/or others. In some embodiments, each instance of a detected visual artifact is associated with a corresponding time range spanning the video frames in which the visual artifact was detected and/or a corresponding confidence level that quantifies the reliability or likelihood of the prediction.

In some embodiments, the feature extraction component(s) 162 includes one or more machine learning models that detect objects or actions from video frames of a video. Any known object or action recognition technique is used to visually extract one or more classes of an object or action from one or more video frames. In an exemplary implementation, one or more neural network classifiers detect the presence of any number of object classes (e.g., hundreds, thousands, and the like) in each video frame. Additionally or alternatively, one or more neural network classifiers detect the presence of any number of action classes (e.g., low-level movement, such as standing up, sitting down, walking around, and talking; higher-level events, such as eating a meal , playing and dancing; and/or other) in sequences of video frames. In some embodiments, each instance of a detected object or action class is associated with a corresponding time range spanning the video frames in which the object or action was detected, a corresponding confidence level that quantifies the reliability or likelihood of a prediction, and/or a or multiple searchable keywords (such as tags) that represent the class.

In some embodiments, feature extraction component(s) 162 includes one or more machine learning models that detect audio events from an audio track associated with video. Any known sound detection technique is used to detect any number of audio event classes (e.g., alarm, laugh, ring, applause, cough, buzz, honk, bark, shoot, siren, and the like). In an exemplary implementation, each frame of audio data from the audio track is encoded (e.g., using linear predictive coding) into a vector representation and classified by one or more neural networks. In an exemplary implementation, consecutive audio frames are grouped together with the same classification and a corresponding time range spanning the audio frames, a corresponding confidence level that quantifies the reliability or likelihood of a prediction, and/or one or more searchable keywords (e.g., tags) identifying the class represent, assigned.

In some embodiments, feature extraction component(s) 162 extracts log events represented in one or more temporal logs, such as software usage logs associated with a video are net. Different implementations imply different types of time logs and/or log events. For example, in an implementation involving screencapture or screencast videos of tutorials for creative software such as ADOBE PHOTOSHOP or ADOBE FRESCO, a software usage log generated by the creative software during screencapturing or screencasting is read to summarize the timing of detected log events, such e.g., tool events (e.g., use of a particular tool, such as selecting a brush, creating a layer, and the like). In an example gaming implementation, a software usage log is read to identify the timing of detected software log events, such as leveling up or defeating an opponent. Although the above examples imply temporal logs with log events derived from video frames, this need not be the case. For example, in an implementation with a live chat or chat stream associated with a live streamed video, a corresponding user chat log or session is read to determine timing of events, such as chat messages about a particular topic , to identify. In an exemplary video streaming implementation (be it live streaming or watching an archived video), a usage log representing how a user/user(s) viewed the video is read to determine the times of detected interaction events, such as navigation events (e.g. play, pause, skip). In general, any type of temporal log and/or metadata can be read to identify log events and corresponding times. In an exemplary implementation, each instance of an extracted log event is associated with a corresponding time range spanning the portion of video in which the log event occurred and/or one or more searchable keywords (e.g., tags) identifying the log event (e.g., for tool events, for a name or action of a software tool).

At the in 1A In the illustrated implementation, video segmentation component 180 executes a segmentation routine that accepts various input parameters, such as identification of designated feature tracks (e.g., predetermined, user selected, and/or other), user requests (e.g., keywords and/or selected facets), minimum or maximum target length of a video segment (which in some cases depends on the zoom level), area of a video to segment and/or others. The video segmentation component 180 uses the input parameters to build a graph with nodes representing candidate intersection points, edges representing candidate segments, and edge weights representing intersection costs, and the video segmentation component 180 computes a video segmentation by solving a shortest path problem to get that path (segmentation) that minimizes the sum of the edge weights (slicing costs).

In some embodiments, one or more aspects of the segmentation routine depend on the type of segmentation (e.g., standard, search, snap point, thumbnail). Depending on the type of segmentation, for example, candidate cut points are determined differently, as will be described in more detail below, different edges are used, different cutting costs are used for edge weights, different minimum or maximum video segment target lengths are used, the zoom level affects the minimum or maximum Video segment target lengths (or not), does a user request affect the segmentation (or not), and/or other dependencies are also possible. Accordingly, in some embodiments, the default segmentation component 164, the search segmentation component 170, the snap point segmentation component 172, and/or the preview image segmentation component 174 trigger the video segmentation component 180 to perform a corresponding video segmentation using corresponding candidate cut points, edges, edge weights, a minimum or maximum video segment target length, a query applicability, and /or other aspects generated. Additionally or alternatively, separate segmentation routines are executed. In an exemplary implementation, a default segmentation is determined at ingestion. In another example implementation, search, snappoint, and/or thumbnail segmentations are determined upon request (e.g., upon receiving a request, loading an agent interface, zooming into a video timeline). However, these are only examples, and other implementations determine the type of segmentation at any appropriate time.

In an exemplary embodiment, the video segmentation component 180 outputs a representation of a complete set (that is, the entire and/or designated area of video) of disjoint (i.e. non-overlapping) video segments and/or timestamps, or any other representation of video segment boundaries (e.g. edit points). In some implementations that determine search segmentation, the output additionally or alternatively includes a representation of whether each video segment in a search segmentation is a match for a particular query (e.g., whether video segments are "on" or "off" with respect to the query. are, a representation of which feature(s) and/or which value(s) of matching segments matched the query, and/or a representation of at what time(s) within each matching segment the query matched .

The embodiment below implies an implementation of a video segmentation component 180 that has one or more common aspects across different types of segmentations. For illustrative purposes, example operations of this implementation of the video segmentation component 180 are described in terms of a standard segmentation and a search segmentation. In this implementation, the video segmentation component 180 includes a candidate boundary selection component 182, a graph building component 184, a slice costing component 186, and a path optimization component 188. Viewed at a high level, the candidate boundary selection component 182 identifies candidate boundaries from the boundaries of feature regions in designated feature lanes. Graph builder component 184 builds a graph with paths representing various segmentation options, slice costing component 186 computes edge weights for edges between nodes in the graph, and path optimizer component 188 solves a shortest path problem along the graph to compute optimal segmentation.

The candidate boundary selection component 182 identifies candidate boundaries from the boundaries of feature areas (feature boundaries) in designated feature tracks as well as within a designated area of the video. In an example standard segmentation, the feature tracks include detected sentences (e.g., from the transcript), detected faces, and detected visual scenes, and the designated area of the video is the entire video. In an example search segmentation, the feature tracks include the same feature tracks as the default segmentation, and the designated area of the video is each video segment in the default segmentation (that is, the search segmentation runs on each video segment in the default segmentation, thereby re-segmenting the default segmentation). In another example search segmentation, the feature tracks include the same or different feature tracks than the default segmentation, and the designated area of the video is the entire video (that is, the search segmentation creates a new segmentation independent of the default segmentation).

In an exemplary implementation, the candidate boundary selection component 182 identifies instances of detected features from the designated feature tracks that overlap the designated area of the video, adjusts feature boundaries to snap to nearby feature boundaries of priority features, and identifies candidate boundaries from the remaining feature boundaries. 2A until 2F are representations of an exemplary technique for computing candidate boundaries from detected feature boundaries. 2A until 2F Figure 12 shows a portion of exemplary designated feature boundaries, the sentence trace 210 and the face trace 220 (where two facets, face1 and face1 and face2, are shown).

2A FIG. 12 is an illustration of an example technique for identifying instances of detected features from the designated feature lanes that overlap with the designated area of video for segmentation. In this example, all detected features from sentence trace 210 and face trace 220 that overlap the designated range between startTime and endTime are identified (the overlapping features are identified by shading). In some embodiments where feature traces are presented with a list of detected instances and corresponding areas, identifying overlapping features includes iterating through the listed areas and generating a representation of those areas that are at least partially aligned with the designated area of video for segmentation overlap.

2 B and 2C 12 are illustrations of an example technique for adjusting feature boundaries for snapping to nearby feature boundaries of priority features. For example, in some embodiments, feature traces are prioritized (e.g., in a priority list) based on which categories of features are considered most important or data highest have been determined to have quality. In an exemplary implementation, a prioritized list of feature tracks includes transcript features (e.g., sentences), then visual scenes, and then faces. The candidate limit selection component 182 iterates over the designated feature lanes in order of priority, beginning with the most important feature lane. For each priority feature boundary of an overlapping feature from a priority feature lane, the candidate boundary selection component 182 blends into the priority feature boundary those feature boundaries from other feature lanes that are within a threshold blending distance (e.g., 500 ms) of the priority feature boundary. Additionally or alternatively, the candidate boundary selection component 182 truncates and/or discards feature boundaries that are outside the designated area of the video for segmentation. In 2 B are feature boundaries 224 and 226 from face trace 220 within a threshold mixing distance of priority feature boundary 222 from sentence trace 210 such that feature boundaries 224 and 226 are at priority feature boundary 220, as in FIG 2C shown, snap into place.

In some embodiments, feature boundaries of detected features within a feature lane are blended at nearby feature boundaries that are within a threshold blending distance outside of the detected features to create, for example, a span that includes both detected features. Additionally or alternatively, feature boundaries of detected features within a feature trace are blended at nearby feature boundaries that have been predicted to have a higher confidence level and are within a threshold blending distance. As in 2 B is shown, feature boundary 232 (for face2 or face2) from face track 220 is within a threshold mixing distance of feature boundary 234 (for face1 or face1) from face track 220 such that feature boundaries 232 are at feature boundary 234, as in FIG 2C shown, snap into place.

2D Figure 12 is an illustration of an example technique for identifying candidate boundaries for a standard segmentation from the remaining feature boundaries. In some embodiments, candidate boundary selection component 182 evaluates the remaining feature boundaries after an adjustment, determines whether the distance between two consecutive feature boundaries is greater than a threshold gap duration (e.g., 500 ms), and if so, adds one or more (e.g., equally spaced ) Bounds into the gap such that the distance between two consecutive feature boundaries is less than the threshold gap duration. In some embodiments, to improve performance, the candidate boundary selection component 182 divides the designated area of the video into a number of equal-length intervals (three in 2D ) and for each interval selects the top N bounds in that interval based on a score function. In an exemplary implementation, a clipping cost is calculated for each of the remaining feature boundaries (to be described later in connection with the clipping cost calculation component 186), and the candidate boundary selection component 182 selects the top N boundaries with the lowest boundary clipping cost.

2E and 2F are representations of an exemplary technique for identifying candidate boundaries for search segmentation from the remaining feature boundaries. In an example search embodiment, a user enters a query that includes one or more keywords and/or selected facets (as described in more detail below). The candidate boundary selection component 182 searches for detected features with associated text or value (e.g., a transcript, object or action tags, audio event tags, log event tags, and the like) that match the keyword(s), using either an exact match and/or a approximate (fuzzy) match. In an exemplary faceted search, a user selects a particular face, speaker, audio class (e.g., music, speech, other), visual scene, visual artifact, and/or other facet, and candidate boundary selection component 182 searches for detected instances of the selected facet(s). 2F 12 shows an example query (“Hello”) with matching features 212 and 214 from sentence track 210. In an example implementation, candidate boundary selection component 182 selects candidate boundaries as those remaining feature boundaries that are within a certain threshold distance (e.g., 5 s) of a boundary of a matching Feature (query hit limit) are located.

As again in 1A As shown, the graph builder component 184 builds a graph with paths representing various candidate segmentations. 3A until 3C 10 are representations of an exemplary technique for building a graph with paths representing various candidate segmentations. In an exemplary implementation, candidate boundaries are defined by Kno ten (for example, nodes 350 in 3A ) connected to edges (such as edge 365a in 3B ) are connected, which in turn are candidate segments (e.g. video segment 365a in 3B ) represent. The different paths through the graph from startTime to endTime represent different candidate segmentations. In 3C For example, edges 362a, 365a, and 368a form a path representing segments 362b, 365b, and 368b. In some embodiments, to support the generation of non-overlapping segmentation (e.g., paths that move forward in time), the graph is constructed as a direct graph, with edges connecting only nodes that move forward in time, or with infinite edge weights moving backwards in time.

In some embodiments, the cutting cost calculation component 186 calculates from 1A Edge weights for the edges in the graph. In an exemplary embodiment, an edge weight for an edge representing a candidate segment quantifies a penalty or cutting cost for the candidate segment (e.g., in a normalized range, such as from 0 to 1 or from -1 to +1). In one non-limiting example, the slice cost calculation component 186 determines the edge weight between two edges as the (e.g., normalized) sum of boundary slice cost and interval slice cost for the candidate segment. In some embodiments, the boundary cutting cost penalizes a candidate boundary for being at a "bad" cutting point (e.g., within a detected feature from another feature trace), and the interval cutting cost penalizes a candidate segment based on negative properties of the candidate segment's span (e.g., having a Length outside of the minimum or maximum target length, an incoherence of overlapping features from other feature tracks, an overlap of both the request ON and OFF features). In some embodiments, the boundary cutting cost for the candidate segment is the boundary cutting cost for the leading boundary of the candidate segment (e.g., when candidate segmentations are complete, candidate segments in one candidate segmentation are therefore adjacent, and the trailing boundary in the next segment is therefore counted).

In an exemplary standard segmentation, the cutting cost calculation component 186 penalizes cutting in the middle of detected feature regions (e.g., sentences, face appearances, visual scenes), determines boundary cutting costs differently relative to different feature lanes, and/or calculates total boundary cutting costs as a combination (e.g as a sum (weighted sum) of separate contributions from each feature lane. In a non-limiting example with designated features including transcript features, visual scenes, and faces, the border cutting cost for a border is calculated as follows. $$\begin{array}{l}bO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst=(3.0*t\mathrm{right}ansc\mathrm{right}iptBO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst+2.0*\\ vis\mathrm{and}alSceneBO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst+faceBO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst)\text{/}6.0\end{array}$$

4A until 4C are representations of an exemplary technique for calculating marginal cutting costs based on visual scenes. 4A until 4C 12 show an example trace of visual scene 405 with different visual scenes represented with different patterns. In some embodiments, a good candidate boundary is (1) close to a visual scene boundary and/or (2) that visual scene boundary has high transition reliability. In an example implementation that considers a candidate boundary related to the visual scene 405 lane, the cut cost calculation component 186 identifies the closest visual scene boundary from the visual scene 405 lane ( 4A ) and calculates the distance to the nearest boundary relative to the duration of the contained visual scene 410 ( 4B ). At the in 4B In the illustrated example, the distance to the nearest boundary (1000ms) relative to the duration of the included visual scene 410 (5000ms) is 1000/5000=0.2. In some embodiments, the slice cost calculation component 186 retrieves the transition reliability of the nearest boundary (0.9 in the example of FIG 4B ) and calculates the boundary cutting cost based on the transition reliability of the nearest boundary, for example: $$\begin{array}{l}vis\mathrm{and}alSceneBO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst=\\ -t\mathrm{right}ansitiOnCOnfi\mathrm{i.e}ence*\left(1.0-2*\text{Relative distance to the border}\right)\end{array}$$

At the in 4B example shown applies: $$vis\mathrm{and}alSceneBO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst=0.9*\left(1.0-2*0.2\right)=-\mathrm{0.54.}$$

4C Figure 12 shows a special case where the included scene 420 has a candidate boundary where the closest visual scene boundary is at an endpoint of the designated area of the video. In some embodiments, when the transition reliability of the closest visual scene boundary is undefined or zero, the cutting cost calculation component 186 calculates a boundary cutting cost based on the distance to the closest boundary relative to the duration of the included visual scene, for example as follows: $$\begin{array}{l}vis\mathrm{and}alSceneBO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst=\\ -\left(1.0-2.0*\left(\text{Relative distance to the n}\ddot{\text{a}}\text{closest visual scene boundary}\right)\right)\end{array}$$

If the relative distance to the nearest visual scene boundary is 0 (the boundary is at the end point of the designated area), Equation 3 solves to -1.0. If the relative distance to the nearest visual scene boundary is 0.5 (the candidate boundary is at the center of the candidate segment), then Equation 3 solves for +1.0.

5 Figure 12 is an illustration of an exemplary technique for computing a marginal cutting cost based on diarized transcript features. 5 Figure 5 shows an example transcript trace 505 with various transcript features diarized according to a detected speaker (showing different speakers with different patterns). In an exemplary implementation, the cutting costing component 186 penalizes candidate boundaries that are within a speech segment (e.g., word segments, phrase segments), candidate boundaries that are during the appearance of a speaker but between sentences, and/or candidate boundaries at the end of a speech segment from which unlikely is that it concludes a topic-related or topical segment. In one embodiment, the editing costing component 186 determines whether a transcript and/or a diarized transcript is available. If transcript features are not available, then there is no contribution to the marginal cutting cost based on the transcript features. At the in 5 In the example implementation shown, where the clipping cost is normalized between -1 (good) and +1 (bad), if a candidate boundary 510 that is in the middle of a word is evaluated with a high clipping cost (e.g., infinite), a candidate boundary 520 will be evaluated that is in the middle of a sentence but between words is scored with an intermediate cutting cost (e.g., 0), a candidate boundary 530 that is in the middle of a speaker but between sentences is scored with a cutting cost equal to or proportional to a predicted probability of the preceding speech segment closing a topical segment (e.g. -0.2) and/or will a candidate boundary 540 located at a change of speakers with a low clipping cost (e.g. -1.0 ) rated. Additionally or alternatively, a candidate boundary at the end of a language segment (e.g., candidate boundaries 530, 540) is scored with a cutting cost equal to or proportional to a predicted probability that the preceding language segment is a topical segment (e.g., -endOfSegmentScore for the previous sentence in 5 ) decides.

6 12 is an illustration of an example technique for calculating a boundary clipping cost based on detected faces. 6 12 shows an exemplary face trace 220 with detected feature regions for each occurrence of a detected face. In an exemplary implementation, the slice costing component 186 penalizes candidate boundaries when the candidate boundaries are within a feature range for the appearance of a detected face. By way of non-limiting example, candidate boundaries that are within the appearance of a detected face (e.g., boundaries 610 and 620) are evaluated with a high clipping cost (e.g., +1.0) and candidate boundaries that are not within the appearance of a detected face (e.g., the limit 640) are evaluated with low clipping costs (e.g., -1.0).

In an example search segmentation, a "good" boundary is at a transition between a feature that matches a query (e.g., a query "IN" feature) and a feature that does not match the query (e.g., a query "OFF". "-Feature). As such, in some embodiments, the slice costing component 186 penalizes candidate boundaries that are within a matching feature (e.g., a query-ON feature) and/or candidate boundaries that are far from a query-ON/OFF transition. 7A until 7D are representations an example technique for calculating a marginal cutting cost based on a request. 7A until 7D 12 show exemplary matching (request-IN) features 710 and 720. In an exemplary implementation, if a candidate boundary is within a matching (request-IN) feature, the clipping cost calculation component 186 evaluates the candidate boundary with a high clipping cost (e.g., +1.0 ). In 7A is the candidate boundary 790 within the matching feature 710, so the clip cost calculation component 186 evaluates the candidate boundary 790 with a high clip cost (of, for example, +1.0).

In some embodiments, when a candidate boundary is not within a matching feature, the cutting cost calculation component 186 considers the two closest and/or adjacent matching features and their lengths. In 7B For example, candidate boundary 794 is outside of matching features 710 and 720 . In an exemplary implementation, the slice costing component 186 considers a window on each side of the candidate boundary 794 (e.g., a window of a size equal to half the average length of the neighboring request-IN features). At the in 7B In the example illustrated, slice costing component 186 determines the size of windows 730a and 730b as 0.5*(5000 ms + 3000 ms)/2 = 2000 ms, where there is a 2000 ms window on either side of candidate boundary 794 . Cost calculation component 186 calculates the amount of request ON time in windows 730a and 730b. In 7C window 730a overlaps with the matching (request-IN) feature 710 for 1800 ms (request-IN time = 1800 ms), and the window 730b does not overlap with the matching (request-IN) feature 720 (request-IN -time = 0 ms). Finally, the cost calculation component 186 calculates the cutting cost based on the amount of overlap of the timeslots 730a and 730b with a matching (request-IN) feature, for example as follows: $$Q\mathrm{and}e\mathrm{right}yBO\mathrm{and}n\mathrm{i.e}a\mathrm{right}yC\mathrm{and}tCOst=1.0-2*\left|\frac{\text{queryOnTimeBefore}-\text{queryOnTimeAfter}}{\text{windowSize}}\right|$$

At the in 7C For example, Equation 4 solves to 1.0-2*1800/2000 = -0.8. 7D FIG. 14 shows another example where a candidate boundary 796 is not within a query-IN feature. In this example, candidate boundary 796 is centered on a query-OFF feature, windows 740a and 740b overlap with a matching (query-IN) feature for 500 ms each, and Equation 4 solves for 1.0-2*0 /2000 = 1.0.

Exemplary interval cutting costs will now be considered. In some embodiments, the clipping cost calculation component 186 assigns an interval clipping cost that penalizes a candidate segment for having a length outside of the minimum or maximum target length, an incoherence of overlapping features from other feature tracks, an overlap of both query IN and OUT features, and /or has other properties. In an exemplary implementation, the slice cost calculation component 186 calculates a total interval slice cost as a combination (e.g., sum, weighted sum) of separate contributions from separate terms (e.g., based on target length, incoherence, partial query matching, and the like).

In some embodiments, the slice cost calculation component 186 calculates an interval slice cost for candidate segments that have a length outside of the minimum or maximum target length. In some embodiments, the minimum or maximum target length for a particular type of segmentation is fixed (e.g., a target segment length of 15s to VideoDuration/5), proportional to, or otherwise dependent on an input zoom level (e.g., the zoom level for a composite timeline in an editor interface, which is explained in more detail below), is presented by an interaction control (e.g., a slider or button that allows a user to set or adjust the minimum or maximum target length), is set to discrete or continuous values mapped and/or other. In an exemplary implementation, a search segmentation has fixed minimum and maximum target segment lengths that are unaffected by zoom level. In another exemplary implementation, a snap point segmentation has minimum and maximum segment target lengths that map to zoom level, where as a user zooms in (e.g., into a composite timeline in an editor interface), the minimum and maximum segment target lengths are reduced, thereby increasing more Provide snap points for smaller video segments.

Depending on designated and/or determined minimum and maximum segment target lengths, the cutting cost calculation component 186 penalizes candidate segments outside of the target length. In some embodiments, the cutting cost calculation component 186 uses a hard constraint in which candidate segments outside the target length are assigned an infinite cutting cost. In some embodiments, the slice cost calculation component 186 uses a soft constraint in which candidate segments outside the target length are assigned a high slice cost (e.g., 10, 100, 1000, and the like).

In some embodiments, the slice cost calculation component 186 calculates an interval slice cost for a candidate segment based on an incoherence of overlapping features from other feature lanes. In some cases, a "good" video segment contains coherent or similar content with respect to each feature track. As such, in an exemplary implementation, the slice costing component 186 penalizes candidate segments that lack coherence in an overlapping region of another feature track (e.g., the detected features when changing the overlapping region). In some embodiments, the slice cost calculation component 186 calculates an interval slice cost based on an incoherence for each feature lane, calculates an interval incoherence slice cost differently for different feature lanes, and/or calculates a total interval incoherence slice cost by combining the contributions from each feature lane (e.g., sum, weighted sum, and the like).

8th 12 is an illustration of an example technique for calculating an interval cutting cost based on an incoherence of detected features. In an exemplary implementation, the slice cost calculation component 186 calculates an interval incoherence slice cost for a candidate segment relative to a feature track based on a number of detected feature transitions in an area of the feature track that overlaps the candidate segment. In 8th For example, candidate segment 810 overlaps two transitions in face track 220 (feature boundaries 855 and 865) such that candidate segment 810 includes sub-segment 850 (with face1 or face1 and face2 or face2), sub-segment 860 (with no faces), and sub-segment 870 (with Face2 or face2). In contrast, candidate segment 820 overlaps with zero transitions in face track 805, so candidate segment 820 only includes sub-segment 880 (having face1 or face1 and face2 or face2). In an exemplary implementation, the slice cost calculation component 186 counts the number of detected feature transitions in the region of a feature track that overlaps each candidate segment, normalizes the number of transitions (e.g., to the maximum number of transitions related to that feature track), and assigns the normalized number of transitions as an interval incoherence cutting cost for a candidate segment.

In some embodiments, some transitions (e.g., feature boundaries) are assigned a measure of the strength of a transition (e.g., endOfSegmentScore, which quantifies a probability that a previous speech segment will conclude a topical segment, a confidence level that a speech segment will be replaced by a new speaker spoken, a measure of the similarity of frames in a visual scene). As such, in some cases the slice costing component 186 penalizes a candidate segment based on a count of overlapping feature transitions weighted by the strength of a transition. This can serve to reduce the incoherence cutting cost based on incoherence of detected features, for example where the sentence changes but the themes are similar or where the visual scene changes but still looks similar.

In some embodiments, slice cost calculation component 186 calculates an interval slice cost for a candidate segment based on a request. In some cases, a "good" video segment is either exclusively ON or exclusively OFF in relation to the query (e.g. to support clean search results). In some implementations, if a user asks for an "elephant," for example, the segments that are returned ideally contain all elephants, or no elephants at all. As such, in an exemplary implementation, the slice costing component 186 only penalizes candidate segments that are partially ON and partially OFF (segments that have elephants for some time). 9A and 9B 10 are illustrations of an example technique for calculating an interval cutting cost based on a query. In 9A candidate segments 910 and 920 are partially ON and partially OFF, and in 9B candidate segments 930 and 940 are all ON or all OFF. With an example In the implementation, the cut cost calculation component 186 counts the number of transitions between ON and OFF segments and normalizes as described above. In another example, the cutting cost calculation component 186 calculates the percentage of overlapping request ON time to total time in the candidate segment, overlapping request OFF time to total time in the candidate segment and assigns lower cutting costs the closer the ratio is to 0 or 1 is.

In summary, this means, as in 1A shown is that the clip cost calculation component 186 calculates clip costs for various candidate segments based on a boundary and/or interval contributions and assigns the clip costs for a candidate segment as an edge weight for the edge representing the candidate segment. Then, the path optimization component 188 solves a shortest path problem along the graph to compute an optimal segmentation using, for example, dynamic programming. In an exemplary implementation, candidate paths are selected to create a complete and non-overlapping segmentation. For each candidate path, the path optimization component 188 sums the edge weights and takes the path with the lowest sum as the optimal path, representing the optimal segmentation.

The foregoing discussion implies an example implementation of the video segmentation component 180 in which the default segmentation component 164 initiates the computation of an example default segmentation and the search segmentation component 170 initiates the computation of an example search segmentation. Another example video segmentation is snap point segmentation, which identifies the locations of selection snap points for a video. As discussed in more detail below, an example use of snap point segmentation in a user interface is with a video timeline for displaying a video (e.g., with a composite timeline representing selected video segments in an editor interface), where the snap points identified by the snap point segmentation are displayed on the timeline and/or used to snap a selection of video segments when a user drags along the timeline or a corresponding portion of the transcript. In various embodiments, a snap point segmentation is calculated at any suitable time (for example, when a video timeline is displayed, an editor interface is activated, the video segments to be represented by a composite timeline are identified, and/or at any other time ). At the in 1A and 1B In the exemplary embodiment illustrated, video interaction engine 108 (e.g., video editing tool 130) of video editing application 105 communicates with snappoint segmentation component 172 to trigger video segmentation component 180 to compute a snappoint segmentation.

In an exemplary implementation of snap point segmentation, the video segmentation component 180 executes a segmentation routine that accepts various input parameters, such as designated feature tracks (e.g., predetermined, user selected, and the like), a minimum or maximum target length of a video segment (which in some cases depends on the zoom level, from an interaction control presented to a user and the like), the region of a video for segmentation (e.g., each video segment designated for editing is represented by the composite timeline, and the like), and/or other. In some embodiments, video segmentation component 180 computes a separate snap point segmentation for each video segment represented by a composite timeline or otherwise designated for processing.

In an exemplary implementation of snap point segmentation, the video segmentation component 180 executes a segmentation routine using any of the techniques described herein. The candidate boundary selection component 182 of the video segmentation component 180 identifies candidate boundaries as, for example, candidate snap points from the boundaries of feature regions in designated feature tracks. In an exemplary embodiment, when no detected features or feature regions are available, the candidate boundary selection component 182 outputs regularly spaced candidate snap points. If detected features and feature regions are available, the candidate limit selection component 182 considers whether transcript features are available. If transcript features are not available, the candidate limit selection component 182 computes candidate snap points that are regularly spaced (e.g., about 500 ms apart are), and then snaps those points to a nearest feature boundary from one of the designated feature tracks that is within a snap threshold (e.g., 250 ms).

In an exemplary embodiment where transcript features are available, candidate boundary selection component 182 iterates through feature boundaries for transcript features (e.g., words) in order, adding the transcript feature boundaries (e.g., word boundaries) as candidate snap points. Additionally or alternatively, when a gap between consecutive transcript feature boundaries (e.g., to represent word duration and/or gaps between words) is greater than a threshold (e.g., 500 ms), candidate boundary selection component 182 inserts regularly spaced candidate snap points (e.g., about 500 ms apart). are) in the gap. In some embodiments, the candidate boundary selection component 182 snaps the added points to a nearest non-transcript feature boundary from one of the designated feature lanes that is within a snapping threshold (e.g., 250 ms). There are only a few ways of designating candidate boundaries as candidate snap points, and any other technique for identifying candidate snap points may be additionally or alternatively implemented, including the other techniques for identifying candidate boundaries described herein.

In some embodiments, the graph building component 184 of the video segmentation component 180 builds a graph with nodes representing the candidate snap points, edges representing candidate separations between snap points (e.g., candidate segments), and edge weights calculated by the slice cost calculation component 186 of the video segmentation component 180. In an exemplary implementation, the slice cost calculation component 186 assigns a slice cost to a candidate segment that promotes snapping to "good" points and/or does not promote snapping to "bad" points. In one non-limiting example, the slice cost calculation component 186 determines the edge weight between two nodes as the (e.g., normalized) sum of a boundary slice cost (e.g., such as described above) and an interval slice cost for the candidate segment. In some cases, with interval cutting costs in mind, snap points that are too obvious may not be helpful. As such, in an exemplary embodiment, a minimum target length between snap points (represented, for example, by a minimum video segment target length) is determined based on minimum pixel separation, which in some cases depends on the zoom level when viewing the video timeline. For example, a designated minimum pixel separation (e.g. 15 pixels) is mapped to a corresponding duration on the timeline (e.g. based on an active zoom level) and this duration is used as the minimum target interval between snap points. The minimum target interval is used in some cases as a hard constraint (e.g., candidate segments shorter than the minimum interval are assigned an infinite interval cutting cost), as a soft constraint (e.g., candidate segments shorter than the minimum interval are assigned a high interval cutting cost of e.g. assigned 10, 100, 1000 and the like) or other.

As such, the slice cost calculation component 186 of the video segmentation component 180 calculates edge weights for edges between nodes in the graph, and the path optimization component 188 of the video segmentation component 180 solves a shortest path problem along the graph to calculate optimal segmentation with resulting segment boundaries representing optimal snap points basis of cutting costs.

Another example video segmentation is thumbnail segmentation, which identifies locations on a video timeline for display with different thumbnails. At an in 1A In the exemplary embodiment illustrated, the preview image segmentation component 174 triggers the video segmentation component 180 to calculate a preview image segmentation. As explained in more detail below, the exemplary use of thumbnail segmentation in a user interface is with a video timeline representing video (e.g., a composite timeline depicting selected video segments in an editor interface), where the thumbnail locations identified by the thumbnail segmentation are included corresponding individual video images (preview images) are displayed on the timeline. In various embodiments, the preview image segmentation is calculated at any given time (for example, when a video timeline is displayed, an editor interface is activated, the video segments to be represented by a composite timeline are identified, and/or at any other time ). At the in 1A and 1B illustrated exemplary embodiment the video interaction engine 108 (e.g., the video editing tool 130) of the video editing application 105 communicates with the preview image segmentation component 174 to trigger the video segmentation component 180 to calculate a preview image segmentation.

In an exemplary implementation of thumbnail segmentation, the video segmentation component 180 performs a segmentation routine using any of the techniques described herein. In some embodiments, video segmentation component 180 performs a segmentation routine similar to the example implementation of snap point segmentation described above, but with the following additional or alternative aspects. For example, the candidate boundary selection component 182 of the video segmentation component 180 identifies candidate boundaries as candidate preview image locations from the boundaries of feature regions in designated feature tracks, the graph builder component 184 of the video segmentation component 180 builds a graph with nodes representing the candidate preview image locations, edges representing candidate separations between preview image locations (e.g., candidate segments ) and edge weights calculated by the edit cost calculation component 186 of the video segmentation component 180 .

In an exemplary implementation, the crop cost calculation component 186 assigns a crop cost for a candidate segment, thereby promoting the placement of thumbnail images in "good" locations. In one non-limiting example, the clipping cost calculation component 186 determines the edge weight between two nodes as a (e.g., normalized) sum of boundary clipping costs (e.g., with penalization of candidate preview image locations that are within a detected feature area or within a portion of video with detected high motion, and the like) and Interval cutting costs for the candidate segment (eg, penalizing candidate thumbnail locations where the visual difference between two consecutive thumbnails is small, penalizing thumbnail separations according to a minimum pixel spacing for a thumbnail, based on a zoom level, and the like).

With regard to boundary clipping costs, in order not to encourage displaying thumbnails at "bad" clipping points (e.g., within a detected visual feature from another feature track), the clipping cost calculation component 186 in some cases assigns low boundary clipping costs to candidate thumbnail image locations based on proximity to a visual feature boundary ( e.g., face, scene), assigns high border clipping cost to candidate preview image locations that are within a detected feature region, and/or assigns high border clipping cost to candidate preview image locations that are within a portion of video with detected high motion (e.g., with detection using one or more machine learning model of feature extraction component / feature extraction components 162).

With respect to interval clipping costs, to encourage displaying thumbnails with different visual content, the clipping cost calculation component 186 in some cases determines interval clipping costs for a candidate thumbnail location based on visual similarity and/or based on differences in visual content between two consecutive candidate thumbnails corresponding to the start - and end boundaries of a candidate separation / segment. In an example implying a face or visual scene transitions, the cutting cost calculation component 186 calculates a measure of the similarity or difference between candidate thumbnails/video frames at thumbnail locations corresponding to the start and end boundaries of a candidate separation/segment and penalizes thumbnail locations where consecutive thumbnails within are of a threshold similarity. Additionally or alternatively, in some cases, thumbnails may be spaced no closer than the width of a thumbnail. In an exemplary embodiment, a target minimum thumbnail separation (e.g., represented by a target minimum video segment length) is determined based on a minimum pixel separation (e.g., a desired thumbnail width), which in some cases depends on the zoom level when viewing a video timeline. For example, a designated minimum thumbnail separation is mapped (e.g., based on an active zoom level) to a corresponding duration on the timeline, which duration is used as the target minimum interval between thumbnails. The minimum target interval is used in some cases as a hard constraint (e.g., candidate separations/segments shorter than the minimum interval are assigned an infinite interval cutting cost), soft constraint (e.g., candidate separations/segments shorter than the minimum interval are assigned a high interval cutting cost). such as assigned 10, 100, 1000 and the like), or other.

As such, the slice cost calculation component 186 of the video segmentation component 180 calculates edge weights for edges between nodes in the graph, and the path optimization component 188 of the video segmentation component 180 solves a shortest path problem along the graph to calculate an optimal segmentation with resulting segment boundaries representing optimal preview image locations basis of cutting costs.

Additionally or alternatively, in some embodiments, the video segmentation component 180 calculates multiple levels of snap point segmentations corresponding to different target video segment lengths (e.g., corresponding to different zoom levels, different input levels set by an interaction control presented to the user, and the like). In some embodiments, lower levels of lockpoint segmentation include lockpoints from higher levels plus additional lockpoints (e.g., the input for a lower-level lockpoint segmentation is a higher-level lockpoint segmentation, the lockpoint segmentation runs on each video segment from the next level up, and so on). These are just a few examples; other implementations are included within the scope of the present disclosure.

In some embodiments, the video segmentation component 180 computes multiple levels of segmentations for a particular type of segmentation (e.g., snap-point segmentation, thumbnail segmentation) corresponding to different zoom levels. For example, when a user zooms in on a video timeline, existing snap points or thumbnails from a higher level segmentation are in some cases included in a lower level segmentation. Similarly, when a user zooms out on a video timeline, snap points or thumbnails from a higher level segmentation are a subset of the snap points or thumbnails from a lower level segmentation. In some embodiments, the graph building component 184 of the video segmentation component 180 builds a graph to support such a hierarchy, with different target video segment lengths being determined for different levels of the hierarchy (e.g., corresponding to different zoom levels, different input levels, the interaction control presented by a user - or control element are set, and the like). As such, in some embodiments, one or more video segmentations are hierarchical in nature.

In some embodiments, the video segmentation component 180 (or any other component) generates a representation of computed video segmentation(s) 196 using one or more data structures. In an exemplary implementation, video segments of video segmentation(s) 196 are identified by values representing or relating to timeline locations (e.g., border locations, identifiers, and the like), segment durations, separations between snap points or thumbnails, and/or other representations. In an exemplary implementation that implies hierarchical segmentation, hierarchical segmentation is represented using a two-dimensional array, where the dimensions of the array correspond to different levels of segmentation, and the values contained in each dimension of the Arrays are stored representing video segments at a corresponding level of the hierarchy.

In some cases, a single copy of a video and a representation of boundary locations are preserved for one or more segmentations. Additionally or alternatively, in an exemplary embodiment implying a particular type of video segmentation of a video file (e.g., standard video segmentation), the video file is segmented at boundary locations of video segments from the (e.g., standard) video segmentation and/or feature boundaries from a feature track (e.g., visual scene boundaries) broken into fragments for efficiency purposes. Users are likely to start or stop playing at boundaries of video segments from a standard video segmentation for motivational reasons. Conventional techniques that generate evenly spaced fragments are likely to require video to start or stop in the middle of a fragment, which in turn results in codec and/or playback inefficiencies. Similarly, evenly spaced fragments are likely to require re-encoding and are therefore more expensive to export. As such, the use of boundaries from one or more video segmentations (e.g., a standard segmentation) and/or feature boundaries from a feature track (e.g., visual scene boundaries) as keyframes to begin a new fragment renders the operations of playing, splicing, and/or or exporting is computationally more efficient.

Interaction with video segmentations

The previous section described exemplary techniques for segmenting a video, for example, in preparation for video editing or other video interactions. By identifying semantically meaningful locations of the video, the video segmentation tool 155 generates a structured representation of the video that provides an efficient and intuitive structure for interacting with the video, for example via the video interaction engine 108 of the video editing application 105 in 1A and 1B provides.

The video interaction engine 108 provides interface functionality that allows a user to select, navigate, play, and/or edit a video through interactions with one or more segmentations of the video and/or detected features of the video. In the exemplary implementation in 1B For example, the video interaction engine 108 includes a video walkthrough tool 110 that provides a viewfinder interface and a video editing tool 130 that provides an editor interface. The video browsing tool 110 (e.g., the viewfinder interface) and/or the video editing tool 130 (e.g., the editor interface) present one or more interaction elements that provide various interaction modalities for selecting, navigating, playing and/or editing a video based on one or more video segmentations 196. In 1 B The video walkthrough tool 110 (e.g., the viewfinder interface) includes various tools such as interactive tiles 112, a selected clips field 114, a default resegmentation tool 116, a search resegmentation tool 118, a transcript tool 120, a segmented timeline tool 122, and a video playback tool 124 . In 1B The video editing tool 130 (e.g., the editor interface) includes various tools, such as a composite clip timeline tool 132, a search resegmentation tool 142, and a video playback tool 144. In various embodiments, these tools are implemented using code that provides a presentation of a corresponding interaction element/ caused by corresponding interaction elements and inputs that interact with the interaction element / the interaction elements, detected and interpreted.

Looking at the video walkthrough tool 110 (e.g., the viewfinder interface), the interactive tiles 112 present the video segments in a standard segmentation and detected features (e.g., the detected features 194 of 1A ) in each video segment. Users can select video segments represented by the interactive tiles 112 and/or the detected features 194 in the interactive tiles 112, jump to corresponding parts of the video, and/or corresponding video segments to that intended for selected clips Add field 114. The default re-segmentation tool 116 recalculates the default segmentation based on selected feature categories (e.g., feature tracks) and updates the interactive tiles 112 to reflect the updated default segmentation. The search re-segmentation tool 118 initiates a search segmentation that re-segments the default segmentation based on a query. Transcript tool 120 presents the portion of the transcript corresponding to the portion of a video displayed in video playback tool 124 . In some embodiments, users can select a portion of the transcript and add a corresponding video segment to the selected clips field 114 . The segmented timeline intended tool 122 provides a video timeline of video that has been segmented based on active segmentation, and the video playback tool 124 plays a selected portion of a video.

With respect to the video editing tool 130 (e.g., the editor interface), the composite clip timeline tool 132 presents a composite video timeline for displaying composite video formed from the video segments selected in the viewfinder interface. In this example, the tool 132 intended for the composite clip timeline includes the feature visualization tool 134, which displays detected features on the timeline, the marquee selection and snapping tool 136, which displays snap points on the timeline and/or snaps a selection to the snap points that Thumbnail preview tool 138, which displays thumbnails on the timeline, and zoom/scrollbar tool 140, which controls the zoom level and position of the timeline. The search re-segmentation tool 142 initiates a search segmentation that re-segments the video segments in the composite video based on a query. The video playback tool 144 plays a selected portion of a video. The editor panel 146 provides any number of editing functions for a selected video segment(s), such as stylistic ones Refinements that transform content, duration affecting refinements that remove content, and/or context functions that depend on the selected content. The functionality of the video walkthrough tool 110, the video editing tool 130, and other exemplary video interaction tools are discussed in more detail below 10 until 15 described.

10 10 is an illustration of an example searcher interface 1000 for traversing standard and/or search segmentations. At the in 10 For example, as illustrated, viewfinder interface 1000 includes a video timeline 1005 (e.g., derived from segmented timeline tool 122 of FIG 1 B controlled), a video frame 1010 (e.g., played by the video playback tool 124 of FIG 1B is controlled), interactive tiles 1020 (e.g., derived from interactive tiles 112 of 1B controlled), a search bar 1060 (e.g., provided by the search resegmentation tool 118 of 1B controlled), a transcript 1080 (e.g., generated by the transcription tool 120 of 1B is controlled) and a selected clips field 1090 (which differs, for example, from the selected clips field 114 of 1 B is controlled or regulated).

In an exemplary use case, a user loads a video for editing, for example using a file explorer, to identify the location of the video (not shown). In some cases, upon receiving a command to load the video to generate one or more segmentations (e.g., from video incorporation tool 160 and/or video segmentation component 180 of 1A ) is ingested and a default segmentation is loaded. In an exemplary implementation, when a video is loaded and/or a user opens the viewfinder interface 1000, the viewfinder interface 1000 presents the video timeline 1005 representing active video segmentation, such as standard segmentation (e.g., by displaying segment boundaries as an underlay). , and/or viewfinder interface 1000 presents a visual overview of a video using interactive tiles 1020 (e.g., interactive tiles 112 of FIG 1 ) that represent video segments in an active video segmentation, such as a standard segmentation. In an example implementation, a standard segmentation is computed from detected sentences, faces, and visual scenes, and the interactive tiles 1020 are arranged in a grid of rows and columns.

In some embodiments, the searcher interface 1000 includes one or more interaction elements (e.g., provided by the standard re-segmentation tool 116 of 1B controlled) presenting one or more input parameters to the default segmentation, thereby allowing a user to change the visual overview (e.g., by designating one or more feature traces to the default segmentation). Exemplary visual reviews include a human-based review (e.g., based on detected faces, detected speakers, and/or detected transcript features), a visual-scene based review (e.g., based on detected visual scenes and/or detected transcript features), an audio-based review descriptive overview (e.g. based on detected audio classifications and/or detected transcript features). Based on one or more designated feature tracks and/or overviews, the default segmentation is recalculated and the interactive tiles 1020 are updated to reflect the updated default segmentation. As such, re-segmentation of the default segmentation allows a user to quickly visualize a video in various ways.

In the viewfinder interface 1000, a user can swipe the video timeline 1005 (which updates the video frame 1010 of the video), scan the transcript 1080, or browse the interactive tiles 1020. Each of the interactive tiles 1020 (e.g., the interactive tile 1030) includes a preview image (e.g., the preview image 1032 of the first video frame of the video segment displayed by the interactive tile 1030) and a representation of one or more detected features and/or a corresponding feature area, such as a transcript (e.g., transcript 1034) from the beginning of that video segment, detected faces (e.g., faces 1036) from that video segment, and one or more of its own faceted timelines (e.g., visual scenes timeline 1038, faceted audio timeline 1040 ). In some embodiments, a faceted timeline represents the detected facets in a particular category of detected features (e.g., visual scenes, audio classifications) and their respective locations in that video segment. Each of the interactive tiles 1020 allows the user to navigate through the video by clicks on one of the facets on the faceted timeline, causing the video frame 1010 to jump to a corresponding part of the video is done. In some embodiments, a user may customize the visualized features in the interactive tiles 1020 by toggling on/off the visualizations for a particular category of feature (e.g., by clicking buttons 1062, 1064, 1066, or 1068 to turn on the visualizations for humans, to control or regulate sounds, visual scenes or visual artifacts).

11A and 11B 11 are representations of example interactive tiles 1110 and 1150. Interactive tile 1110 includes a thumbnail 1115 of the first video frame of the video segment represented by interactive tile 1110, a transcript 1120 from the beginning of that video segment (represented by placeholder text), a timeline intended for visual scenes 1125, a faceted audio timeline 1140, the clip duration, and an add button 1148 for adding the video segment represented by the interactive tile 1110 to a selection. In this example, the timeline 1125 intended for visual scenes is faceted based on the instances of detected visual scenes that appear in the video segment represented by the interactive tile 1110 . In particular, segment 1130 represents a visual scene, while segment 1135 represents another scene. In addition, the faceted audio timeline 1140 is faceted based on the instances of detected audio classifications (e.g., music, speech, other) that appear in the video segment represented by the interactive tile 1110 . In particular, segments 1144 represent one audio classification (e.g., speech), while segments 1142 represent another audio classification (e.g., music). In this example, a user can click on one of the facets from a faceted timeline (e.g. segments 1130, 1135, 1142, 1144) to jump to that part of the video. In another example, the interactive tile 1150 includes a representation of the detected faces 1160 in the video segment represented by the interactive tile 1160 . Visualizations of detected features like these help the user to step through the video without playing the video.

In some embodiments, hovering over a portion of an interactive tile, such as a faceted timeline, a thumbnail, and/or elsewhere in the interactive tile, updates the thumbnail in the interactive tile or presents a pop-up with a thumbnail (e.g., a pop -up thumbnail 1055 of 10 ) from a relevant part of the video. The horizontal input position (for example, the x-position of a mouse or click input) relative to an overall width (for example, a faceted timeline, thumbnail, interactive tile to hover over) is mapped to a percentage offset into that represented by the interactive tile video segment is mapped and a corresponding thumbnail is looked up and displayed. In some embodiments, evenly sampled video frames are available as thumbnails. In other embodiments, the available thumbnails are identified by thumbnail segmentation (e.g., outputting the closest available thumbnails or the available thumbnails within a threshold distance of the horizontal input position). As such, the user may swipe through a set of thumbnails by hovering over one or more portions of an interactive tile.

At the in 10 In the illustrated embodiment, searcher interface 1000 includes a search bar 1060. In this example, search bar 1060 takes a query in the form of one or more keywords (e.g., entered into search field 1070) and/or one or more selected facets (e.g., provided via appropriate menus accessed via buttons 1062, 1064, 1066 and 1068) and triggers a search segmentation to re-segment the default segmentation based on the query. In an example implementation, a user types one or more keywords into the search field 1070 and/or selects one or more facets via menus or other interaction elements representing feature categories (feature traces) and/or corresponding facets (detected features). In 10 interacting with buttons 1062, 1064, 1066, and 1068 (e.g., by hovering over the menu, left-clicking on a button corner, right-clicking) activates a corresponding menu that includes detected people (button 1062), detected sounds (button 1064), detected visual scenes (button 1066) and detected visual artifacts (button 1068). 12A until 12D are representations of example faceted search menus. 12A shows an example menu of detected faces (e.g., indicated by button 1062 of 10 is activated), 12B shows an example menu with detected tones (e.g. activated by button 1064 of 10 is activated), 12C shows an example menu with detected visual scenes (e.g. displayed by button 1066 of 10 is activated), and 12D shows an example menu of detected visual artifacts (e.g., displayed by button 1068 of 10 is activated). As such, a user can through the in 12A until 12D navigate the faceted search menus shown, select one or more facets (e.g., a specific face, sound classification, visual scene, and/or visual artifact), enter one or more keywords in the search field 1070, and perform a search (e.g., by clicking on a facet, on an existing faceted search menu, click on a search button, and the like).

In some embodiments, a typed keyword search triggers a search for detected features with associated text or value (e.g., a transcript, object or action tags, audio event tags, log event tags, and the like) matching the keyword and/or a selected facet triggers a search for detected instances of the selected facet/facets. In an exemplary implementation, search bar 1060 triggers search segmentation component 170 and/or video segmentation component 180 of 1A computes a search segmentation that matches that of the interactive tiles 1020 of FIG 10 represented standard segmentation is re-segmented, thereby updating the interactive tiles 1020 to represent the video segments of the resulting search segmentation. In this example, searching is for breaking, based on a query, the interactive tiles that match the query to represent smaller units of video. In other words, tiles that match the request are broken up into smaller video segments, while tiles that don't match are left as is. In some embodiments, tiles that match the query and are broken up into smaller video segments are animated, for example showing how the tile is broken up.

In some embodiments, viewfinder interface 1000 highlights interactive tiles representing matching video segments (request-IN segments). The interactive tile 1150 from 11b is shown with a segment 1170 indicating that the tile is a hit for the query. Other examples of highlighting include highlighting, adding a fill (such as a transparent fill), and/or others. In some embodiments, a matching tile additionally or alternatively represents why the tile is a hit, for example by presenting or highlighting a representation of the feature(s) that matched the query (e.g. a matching face, a matching visual scene, a matching sound classification, a matching object, a matching keyword, and so on). The interactive tile 1150 from 11b For example, highlights one of the faces 1160, indicating that the face matched the query. In another example, a match against the transcript is for highlighting, underlining, or otherwise emphasizing a matching word in the tile. In this way, a user can easily see which interactive tiles match a query and why.

In some cases, when a user searches for content, the size of the video segments that the user desires may vary based on the task. For example, if a user wants to find clips of giggling children, the user may only want short search results of a few seconds. However, if the user wants clips of a rocket launch, the user may want longer search results. As such, in some embodiments, viewfinder interface 1000 provides a slider or other interaction element (not shown) that presents an input parameter for segmentation (e.g., a minimum and maximum target length of the video segment), allowing the user to set the size of video segments generated by the segmentation and rendered by the interactive tiles 1020. In some embodiments, one or more of the interactive tiles (e.g., each tile) provides its own slider or other interaction element (e.g., a handle) that presents an input parameter, allowing the user to specify the size of a video segment / of video segments represented by a particular tile. Various embodiments therefore provide one or more interaction elements that enable a user to break tiles into smaller parts locally (per tile) and/or globally (all tiles).

Transcript 1080 presents the transcript for the video and highlights an active portion 1085 of the transcript. In some examples, transcript 1080 provides a diarized transcript representing a detected speaker of the portion of the active portion of transcript 1085 .

The selected clips field 1090 represents video segments added by a user to a selection. In an example implementation, a user may add video segments to the collection by dragging an interactive tile into the selected clips box 1090, onto the "+" button in an interactive tile (e.g., button 1148 in the interactive tile 1110 of FIG 11A ) clicks, interacts with a visualization of a detected feature or facet in an interactive tile (e.g. by right-clicking on a visualization, for example on one of the segments 1130, 1135, 1142, 1144 or one of the facets 1160 of 11A to activate a context menu and add a corresponding subset of a video segment for selection from the context menu, by highlighting part of the transcript 1080, right-clicking to activate a context menu and adding to selection from the context menu), and/or others. In 10 the Selected Clips panel 1090 displays thumbnails or a list of selected video segments.

Once a set of video segments is selected, a user can switch to an editor interface to perform one or more editing functions. At the in 10 For example, as illustrated, the finder interface 1000 provides one or more navigation elements that navigate between the finder and editor interfaces (e.g., a finder button 1095, an editor button 1097, an editor button in the selected clips field 1090, and the like).

13 Figure 12 is an illustration of an example search in a searcher interface. In this example, a user enters the query "giggle" into the search field 1310, which triggers search segmentation that highlights matching tiles (e.g., matching tile 1320) and matching portions of the transcript (e.g., transcript region 1325). In this example, the user adds appropriate video segments represented by tiles 1332, 1334, 1336 to the selected clips field 1340, causing the added video segments to be displayed with thumbnail images 1342, 1344, 1346. In 13 the user is in the process of adding another matching video segment by clicking on the add button 1130 in a corresponding interactive tile. Once the user has finished adding video segments to the selection in the selected clips field 1340, the user clicks on the button 1350 ("Edit your clips") to switch to an editor interface, such as the one shown in 14 is shown.

14 14 is an illustration of an example editor interface 1400 for video editing. At the in 14 For example, editor interface 1400 includes video timeline 1405 (e.g., derived from composite clip timeline tool 132 of 1B controlled), a search bar 1450 (e.g., provided by the search resegmentation tool 142 of 1B is controlled) and an agent field 1460 (which, for example, is controlled by the agent field 146 of 1B is controlled or regulated). Video timeline 1405 includes thumbnails 1410 (e.g., generated by thumbnail preview tool 138 of 1B controlled), a faceted audio timeline 1420, and a faceted artifact timeline 1430 (e.g., generated by the feature visualization tool 134 of FIG 1B controlled) and a marquee selection 1440 (e.g., created by the marquee selection and snapping tool 136 of FIG 1 B is controlled or regulated).

In an exemplary implementation, editor interface 1400 presents video timeline 1405 that represents active video segmentation (e.g., by displaying segment boundaries as an underlay). In an exemplary use case, a user loads a video for editing, for example using a file explorer, to identify the location of the video (not shown). The video is used to generate one or more segmentations (e.g., by video incorporation tool 160 and/or video segmentation component 180 of 1A ) ingested, and/or the editor interface 1400 initializes the video timeline 1405 with a segmentation of the entire video (e.g., a default segmentation). In another example use case, a subset of the video segments in a video is preselected or otherwise designated for editing (for example, when switching from a viewfinder interface in which one or more video segments have been added to a field intended for selected clips, when loading an existing one processing project). When the editor interface 1400 is opened and/or the existing editing project is loaded, the editor interface 1400 initializes the video timeline 1405 with the designated video segments. In some cases, the video timeline 1405 represents composite video made up of those video segments that need to be edited processing and/or re-segmentation of those video segments (e.g. through search segmentation). At a high level, a user can select, edit, move, delete, or otherwise manipulate video segments from video timeline 1405 .

In an example implementation, the finder and editor interfaces are represented by one or more navigation elements (e.g., the finder and editor buttons 1095 of 1097 of 10 , the viewfinder and editor buttons 1495 of 1497 of 14 ) linked that toggle between the finder and editor interfaces. In this implementation, a user can use the viewfinder interface to step through video and add video segments to a selection, switch to the editor interface, and perform one or more refinements or other editing operations. In some embodiments, when a user designates video segments for editing from the viewfinder and switches to the editor interface, the editor interface creates a composite video representation that arranges the selected video segments in a chronological order, in an order in which they added to the selection, in any designated order (e.g. the order in which they were placed in the fields intended for selected clips) and/or otherwise. In some embodiments, the editor interface 1400 (e.g., the video timeline 1405) depicts the boundaries of the designated video segments and/or depicts detected features in the designated video segments Once video has been added, the user can remove the content that is no longer desired, but can add new content to the composite video, and the user can return to the searcher interface.

In some embodiments of editor interface 1400, a user can swipe video timeline 1405 and jump to different parts of a composite video by clicking the timeline. Additionally or alternatively, a user can jump to different parts of a composite video by scanning the transcript 1445 and clicking on a particular section (e.g., a word). In some embodiments, a transcript is provided to the side of the video on which the video (such as in 14 ) and/or presented elsewhere.

In some embodiments, to help identify particular portions of the composite video, the video timeline 1405 represents one or more detected features and/or where they are located in the composite video (e.g., corresponding feature regions). video timeline 1405 represents each detected feature category with a corresponding faceted timeline representing detected facets (e.g., faces, audio classifications, visual scenes, visual artifacts, objects or actions, and the like) and their respective locations in that video segment. In some embodiments, a user may customize the visualized features on video timeline 1405 by turning on/off the visualizations for a particular category of feature (e.g., by clicking buttons 1462, 1464, 1466, 1468 to control the visualizations for people, sounds, visual scenes or visual artifacts). At the in 14 In the illustrated embodiment, a user has enabled visualizations for sounds (e.g., via button 1464) and visual artifacts (e.g., via button 1468) such that video timeline 1405 includes faceted audio timeline 1420 and faceted artifact timeline 1430. In some embodiments, clicking on a particular facet from a faceted timeline serves to select a corresponding portion of the video (e.g., as represented by marquee selection 1440 of a portion of the video with a selected visual artifact). In this way, a user can easily select and delete those parts of the video with detected visual artifacts, for example. In general, snapping to semantically meaningful snap points assists users in quick trimming.

In some embodiments, a portion of the composite video represented by the video timeline 1405 is selectable through interactions with the video timeline 1405 and/or the transcript 1445. In general, the selection is highlighted in any suitable manner, such as marking it (e.g., with a dashed line), adding a fill (e.g., a transparent fill) to a selected area, and/or other means. In an example implementation, a selection (e.g., a marquee selection, such as marquee selection 1440) is created by clicking or tapping and dragging or tapping on video segments represented in video timeline 1405 or transcript 1445 .Dragging done. In some embodiments, a selection made in an element (video timeline 1405 or Transcript 1445) has been made, additionally highlight (eg, highlight) a corresponding portion of the other element (not shown). In some cases, a selection can be edited after it has been moved away by clicking and dragging (dragging) the start and/or end point of the selection. In an exemplary implementation, a selection drag operation (e.g., along video timeline 1405, transcript 1445) causes selection boundaries to snap to snap points defined by snap point segmentation and/or a current zoom level (calculated as described above). In some embodiments, the video timeline 1405 presents a visualization of the snap points defined by a snap point segmentation and/or a current zoom level. In some cases, the snap points are only displayed during a drag operation (e.g., on the video timeline 1405), so the displayed snap points on the video timeline 1405 disappear when the drag operation is terminated.

15A and 15B are representations of an example marquee selection with a snap. In 15A For example, a user has clicked and dragged left with a cursor 1510, creating a marquee selection 1520 that snaps to snap points represented with vertical bars (e.g., snap points 1530). In this example, the user previously added video segments that included matches for the "giggle" query, so some of the video segments matched based on the detected giggles in the audio track. In 15A and 15B the faceted audio timeline 1540 represents various detected audio classifications (e.g., by representing detected music with one color, detected speech with a second color, and other sounds with a third color), as well as the volume levels of the audio track in various ways 15B the faceted audio timeline 1540 represents detected giggles 1550, 1560, 1570 as different tones. However, the detected giggles 1560 are from a video segment with a detected visual artifact 1580. To remove this clip, the user moves the marquee selection 1520 to the in 15B location shown, thereby ensuring that parts of the video with the other detected giggles 1550 and 1570 are omitted. As such, the user can select the video segment within the marquee selection 1520 in 15B delete to remove the video segment with the detected visual artifact 1580 from the composite video.

As in 14 As shown, in some embodiments, video timeline 1405 presents one or more thumbnail images (e.g., thumbnail images 1410). In an example implementation, when a user first opens editor interface 1400, editor interface 1400 (e.g., video timeline 1405) presents each video segment in the video with a thumbnail at the beginning of each video segment. Additionally or alternatively, editor interface 1400 (e.g., video timeline 1405) Display thumbnails at locations of the video identified by thumbnail segmentation (calculated as described above). In an exemplary implementation, each video segment includes at least one thumbnail, with longer video segments being more likely to include multiple thumbnails. In some embodiments, when a user zooms into the video timeline 1405, more thumbnails appear (e.g., based on a thumbnail segmentation calculated for that zoom level) and/or thumbnails visible at a higher zoom level remain. As such, thumbnails on the video timeline 1405 serve as landmarks that aid in navigating through video and selecting video segments.

In some embodiments, video timeline 1405 includes a zoom/scroll bar tool (eg, derived from zoom/scroll bar tool 140 of FIG 1B controlled) that allows the user to change the zoom level and scroll to different positions on the video timeline 1405 . In some cases snap points, thumbnails and/or visualized features are zoom level dependent. In an example implementation, when a user zooms into a video timeline, different snap point segmentations and/or thumbnail segmentations are calculated or looked up based on the zoom level (e.g., different zoom levels map to different values of one or more segmentation parameters, such as video segment target lengths, a minimum target interval between snap points, a minimum target interval between thumbnails). Additionally or alternatively, the editor interface 1400 provides one or more interaction elements presenting one or more segmentation parameters for a snap point segmentation and/or a thumbnail segmentation, thereby giving the user the ability to control the granularity of the snap point segmentation and/or the thumbnail segmentation regulate. In some embodiments, zooming in and out of the video timeline 1405 expands and collapses the detail of visualized features on the video timeline 1405. In an exemplary implementation, a high zoom level displays an audio facet timeline representing MSO (Music/Speech/other MSO, music/speech/other) audio classifications, and zooming in adds a representation of detected audio events to the video timeline 1405. Additionally or alternatively, higher zoom levels support feature regions or other feature visualizations that are expanded to show more detail at lower zoom levels.

In some embodiments, the agent interface 1400 accepts a query (e.g., a keyword and/or a facet), triggers a temporary search segmentation that segments the video segments in the composite video based on the query, and presents a visualization of the search segmentation (e.g., by rendering the boundaries of the video segments thereof as an underlay to the video timeline 1405). At the in 14 In the illustrated embodiment, agent interface 1400 includes search bar 1450. In this example, search bar 1450 takes a query in the form of one or more keywords (e.g., entered into search field 1470) and/or one or more selected facets (e.g., entered via appropriate menus accessible via buttons 1462, 1464, 1466, and 1468) and initiates a search segmentation that re-segments the video segments in the composite video based on the query. In an example implementation, a user types one or more keywords into the search field 1470 and/or selects one or more facets via menus or other interaction elements that represent feature categories (feature traces) and/or corresponding facets (detected features). In 14 interacting with buttons 1462, 1464, 1466, and 1468 (e.g., by hovering over the menu, left-clicking on a button corner, right-clicking) activates a corresponding menu showing detected people (button 1462), detected sounds (button 1464), detected visual scenes (button 1466) and detected visual artifacts (button 1468). As such, a user may navigate through the faceted search menus, select one or more facets (e.g., a particular face, sound classification, visual scene, and/or visual artifact), enter one or more keywords into the search field 1470, and enter a Perform a search (e.g., by clicking a facet, an existing faceted search menu, clicking a search button, and the like).

In some embodiments, when a user makes a query through a facet or keyword, search bar 1450 triggers temporary search segmentation and highlights the matching video segments in video timeline 1405 . In this example, the search segmentation is considered temporary because it does not perform any destructive operations on the video segments in the composite video. If the user makes another query by adding or removing keywords or facets, the search bar 1450 triggers a new temporary search segmentation. If the user deletes or removes the query, the search segmentation disappears and the video timeline 1405 retrieves the composite video representation as it was before the search. In some embodiments, keyword and facet queries persist until a user deletes or removes them, thereby de-emphasizing any search result. In an exemplary implementation, the search state does not persist when the user toggles between the searcher and editor interfaces.

In some embodiments, a search segmentation in the editor interface 1400 considers any existing video segments in the composite video without changing any boundaries thereof (e.g., the search segmentation is performed on each video segment in the composite video separately). When a user performs an action on a search result (e.g., deletes a matching video segment), new edit boundaries reflecting the action are created. In other words, if a user searches for "tree" and deletes a video segment that shows a tree in one part of the composite video but not another part, the part where the user did the operation (in this case, the deletion) has performed new segment boundaries, while this is not the case for the other parts. In some embodiments, search results are affected by the zoom level (e.g., shown in more detail or detail with the zoom level) and/or corresponding areas on the video timeline 1405 are rendered to show where in the composite video the query is ON.

Various embodiments of the video timeline 1405 therefore present a high-level overview of visual and audio content contained in a composite video, depending on the toggled on feature categories, the zoom level of the search criteria, and the screen size. As such, in different embodiments, a user can simultaneously view detected features, snap points, video thumbnails and/or search results, which assists the user in choosing good cutting points.

In an example implementation, after selecting one or more video segments, one or more editing functions provided by editor interface 1400 are used to edit the selected video segments. For example, the editor panel 1460 provides any number of editing functions for a selected video segment. Depending on the implementation, available editing features include stylistic refinements that transform content (e.g., wind noise reduction), duration-affecting refinements that eliminate content (e.g., "hide" areas of footage, remove profanity, perform a time warp, trim to n seconds), and/or context functions that depend on the selected content (e.g. remove or beep words from the content with an appropriate transcript). In some embodiments, the refinement of the video properties is declarative and non-destructive. For example, if a marquee selection is made and overlaps portions of the composite video that have previously applied properties, any newly applied properties will overwrite conflicting properties with the new value. In various embodiments, the editor panel 1460 provides any suitable editing functionality, including reordering, cropping, applying transitions or effects (e.g., rate of change, volume), adjusting color, adding captions or graphics, and/or others.

As such, the resulting composite video can be played, saved, exported, or other operations performed. In one example, the video segments in the composite video are played (e.g., upon clicking a play button), skipping video segments that are not in the composite video. In another example, the video segments in the composite video are exported. Depending on the implementation, any known tool or technique is used to perform any type of operation on the video segments in the composite video.

The video segmentation and interaction techniques described above are intended to be mere examples. Other modifications, combinations, and sub-combinations are included within the scope of the present disclosure.

Exemplary flowcharts

In 16 until 22 Flowcharts are shown that represent various procedures. Each block of methods 1600-2200, as well as other methods described herein, includes a computational process performed using any combination of hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a standalone application, service, or hosted service (standalone or in combination with another hosted service), or a plugin for another product, to name a few.

Become first 16A and 16B considered. These figures show a method 1600 for generating a video segmentation using a graphical model according to embodiments described herein. First, at block 1610, one or more feature tracks are accessed. The one or more feature traces represent instances of detected features in a video and corresponding time ranges in which the instances are present in the video. To generate optimal video segmentation, candidate segmentations are evaluated using a graphical model. Specifically, at block 1620, a representation of a graph is generated with nodes and edges such that the nodes represent candidate boundaries selected from time domain boundaries and the edges represent candidate video segments and have edge weights representing cutting costs for the candidate video segments. Blocks 1640 and 1650 show one possible way of performing at least a portion of block 1620.

At block 1640, candidate boundaries are selected from a subset of the boundaries of the time ranges in which the detected features are present in the video. Blocks 1642 to 1646 from 16B show one possible manner of performing at least a portion of block 1640. At Block 1642, instances of detected features are identified from designated feature tracks that overlap a designated area of video for segmentation. These identified features can be thought of as "overlapping features" because they are present in the video during time ranges that overlap with the designated area of the video for segmentation (e.g., the entire video, a specific video segment). At block 1644, boundaries of time ranges in which the identified instances (overlapping features) exist are adjusted to snap to nearest boundaries of time ranges in which priority features exist, thereby forming a subset of the time range boundaries. In an exemplary implementation, a prioritized list of features includes transcript features (e.g., sentences), then visual scenes, and then faces. At block 1646, candidate boundaries are identified from the subset.

At block 1650, a clipping cost for the edge weights in the graph is calculated. Blocks 1652 to 1656 from 16B 16 show one possible manner of performing at least a portion of block 1650. At block 1652, a boundary cutting cost is determined for a candidate segment. In an exemplary embodiment, the boundary cutting cost penalizes the leading boundary of the candidate segment based on negative properties of the leading boundary. In some embodiments, the boundary clipping cost penalizes the candidate segment's leading boundary for being within a detected feature from another feature track (e.g., a detected word) based on a distance to a detected visual scene boundary from the boundaries of the time domains or in other scenarios. In some embodiments, a boundary cutting cost is calculated using Equation 1. At block 1654, interval cutting costs are determined for a candidate segment. In an exemplary embodiment, the interval cutting cost penalizes a candidate segment based on negative properties of the candidate segment's span. In some embodiments, the interval cutting cost penalizes candidate segments that have a length outside of a minimum or maximum target length, candidate segments that have an incoherence of overlapping features from other feature lanes, and/or candidate segments that overlap with both query IN and OFF features. At block 1656, the edge weight between two nodes is determined as the sum of the boundary cutting cost and interval cutting cost for the candidate segment.

As in 16A As shown, at block 1660 a shortest path through the nodes and edges is computed. At block 1670, a presentation of a representation of a segmentation of the video according to the shortest path is prompted.

17 17 shows a method 1700 for segmenting a standard segmentation of a video into a search segmentation in accordance with embodiments of the present invention. First, at block 1710, a presentation of a default segmentation of video is caused. At block 1720, a request is received. At block 1730, detected features of the video are searched for matching features that match a query. At block 1740, the default segmentation is re-segmented based on the search segmentation request. Block 1745 shows one possible manner of performing at least a portion of block 1740. At block 1745, each video segment of the default segmentation that includes at least one of the matching features is segmented based on the request. At block 1750, the presentation is updated to reflect the search segmentation.

18 18 shows a method 1800 for navigating through a video using interactive tiles according to embodiments of the present invention. First, at block 1810, a presentation of interactive tiles is prompted that represent (i) video segments of a segmentation of video and (ii) instances of detected features of the video segments. At block 1820, a click or tap on one of the interactive tiles is detected. In particular, the clicking or tapping occurs on a visualization of one of the instances of one of the detected features detected from one of the video segments presented by the interactive tile. At block 1830, the video is navigated to a portion of the video where the instance of the detected feature is present.

19 19 shows a method 1900 for adding video segments to a selection according to embodiments of the present invention. First, at block 1910, a presentation of interactive tiles is prompted that represent (i) video segments of a segmentation of video and (ii) instances of detected features of the video segments. At block 1920, an interaction with one of the interactive tiles representing one of the video segments is detected. Blocks 1930 through 1950 show one possible manner of performing at least a portion of block 1920. At block 1930, a click or tap (e.g., a right-click or a tap-and-hold) in the interactive Tile detected. The clicking or tapping occurs on a visualization of one of the instances of one of the detected features detected from one of the video segments presented by the interactive tile. At block 1940, a context menu is activated in response to the click or tap. At block 1950, input selecting an option from the context menu is detected. The option is to add to the selection the portion of the video segment corresponding to the instance of the detected feature. At block 1960, at least a portion of the video segment is added to a selection of video segments in response to detecting the interaction.

20 FIG. 2000 shows a method 2000 for snapping a selection boundary of a selection of video segments, in accordance with embodiments of the present invention. First, at block 2010, a presentation of a first segmentation of a video timeline of video is caused. At block 2020, a representation of a second segmentation of the first segmentation is generated using one or more feature tracks representing instances of detected features in the video and feature regions indicating when the instance is present in the video. At block 2030, in response to a drag operation along the video timeline, a selection boundary of a selection of a portion of the video is snapped to snap points defined by the second segmentation.

21 FIG. 2100 shows a method 2100 for presenting a video timeline with thumbnails at locations defined by thumbnail segmentation, according to embodiments of the present invention. First, at block 2110, a presentation of a first segmentation of a video timeline of video is caused. At block 2120, generation of a representation of a thumbnail segmentation of the video is triggered. Generating the representation of the thumbnail segmentation uses one or more feature tracks that represent instances of detected features in the video and feature regions that indicate when the instances are present in the video. At block 2130, the video timeline is updated to display one or more thumbnails of the video at locations on the video timeline defined by the thumbnail segmentation.

22 Figure 2200 shows a method 2200 for presenting a video timeline with thumbnails at locations defined by thumbnail segmentation. First, at block 2210, a presentation of a video timeline of video is caused. At block 2220, a representation of a thumbnail segmentation of the video is accessed. Thumbnail segmentation defines thumbnail locations on the video timeline at feature region boundaries of times at which instances of detected features of the video are present in the video. At block 2230, the presentation is updated to include a thumbnail at one of the thumbnail locations defined by the thumbnail segmentation, the thumbnail representing a portion of the video associated with the thumbnail location.

Exemplary Operating Environment

Having provided an overview of embodiments of the present invention, an exemplary operating environment in which embodiments of the present invention may be implemented is described below to provide general context for various aspects of the present invention. In particular 23 An exemplary operating environment for implementing embodiments of the present invention is shown in FIG. Computing device 2300 is merely one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Likewise, the computing device 2300 should not be interpreted as having any dependency or requirement associated with any one or combination of the illustrated components.

The invention may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine, such as a cellular phone, personal digital assistant, or other handheld device. Generally, program modules, which include routines, programs, objects, components, data structures, and the like, refer to code that performs specific tasks or implements specific abstract data types. The invention may be practiced in a variety of system configurations, including handheld devices, consumer electronic devices, general purpose computers, specialized computing devices, and the like. The invention can also be implemented in distributed computing environments where tasks are performed by remote processing devices linked through a communications network.

As in 23 As shown, computing device 2300 includes a bus 2310 that directly or indirectly couples the following devices: memory 2312, one or more processors 2314, one or more presentation components 2316, I/O ports (input/output I/O, input /output) 2318, I/O components 2320, and an illustrative power supply 2322. Bus 2310 represents what may be one or more buses (such as an address bus, a data bus, or a combination thereof). Although the different blocks of 23 shown with lines for the sake of clarity, in reality the demarcation of the various components is not so clear-cut, so the lines should be figuratively gray and blurred. For example, a presentation component, such as a display device, can also be considered an I/O component. In addition, processors have a memory. It is recognized for the purposes of the present invention that this is the nature of the field and it is again pointed out that the diagram of FIG 23 is purely illustrative of an exemplary computing device that may be used in connection with one or more embodiments of the present invention. No distinction is made between categories such as "workstation", "server", "laptop", "handheld" and the like, as these are all within the scope of 23 and are covered by the term "Computing Device".

Computing device 2300 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 2300 and includes both volatile and non-volatile media and removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communications media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, and other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to desired information to be stored and accessible by computing device 2300. Computer storage media do not include signals per se. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information distribution media. The term "modulated data signal" means a signal that has one or more properties adjusted or changed in a way that information is encoded in the signal. By way of example and not limitation, communication media includes wired media, such as a wired network or direct wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media. Combinations of any of the foregoing are intended to be included within the scope of the computer-readable media.

Memory 2312 includes computer storage media in the form of volatile and/or non-volatile memory. The memory can be removable, non-removable, or a combination of both. Exemplary hardware devices include solid state memories, hard drives, optical disk drives, and the like. Computing device 2300 includes one or more processors that read data from various entities such as memory 2312 or I/O components 2320 . A presentation component(s) 2316 presents data information to a user or other device. Exemplary presentation components include a display device, a speaker, a printer component, a vibration component, and the like.

I/O ports 2318 allow computing device 2300 to be logically coupled to other devices that include I/O components 2320, some of which may be built-in. Illustrative components include a microphone, joystick, gamepad, satellite dish, scanner, printer, wireless device, and the like. The I/O components 2320 may provide a natural user interface (NUI) that processes air gestures, speech, or other physiological input generated by a user. In some cases, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, pen recognition, face recognition, biometric recognition, Both on and off screen gesture recognition, air gesture, head and eye tracking, and touch recognition (described in more detail below) in conjunction with a display of the computing device 2300. The computing device 2300 may be equipped with depth cameras, such as stereoscopic camera systems , infrared camera systems, RGB camera systems, touchscreen technology and combinations thereof for detecting and recognizing gestures. Additionally, the computing device 2300 may be equipped with accelerometers or gyroscopes that enable motion detection. The output of the accelerometers or gyroscopes can be provided to the display of the computing device 2300 to provide immersive augmented reality or virtual reality.

The embodiments described herein support editing or playing a video. The components described here refer to integrated components of a video editing system. The integrated components denote the hardware architecture and software framework that support the functionality using the video editing system. Hardware architecture refers to physical components and the interrelationships between them, while software framework refers to software that provides functionality that can be implemented with hardware embodied on a device.

The end-to-end software based video editing system can operate within the video editing system components to operate computer hardware which in turn provides the functionality of the video editing system. At a low level, hardware processors execute instructions selected from a machine language (also referred to as machine code or native) instruction set for a given processor. The processor recognizes the native instructions and performs appropriate low-level functions associated with, for example, logic, control, and memory operations. Low-level software written in machine code can provide more complex functionality for higher levels of software. As used herein, computer-executable instructions include any software, including low-level software written in machine code, high-level software such as application software, and any combination thereof. In this context, components of the video editing system can manage resources and provide services for the functionality of the video editing system. Any further modifications and combinations thereof are included in the embodiments of the present invention.

Although some implementations have been described in terms of neural networks, in general, embodiments may be implemented using any type of machine learning model(s), such as linear regression, logistic regression, decision trees, support vector machines (SVM ), naive Bayes, k-nearest neighbor (Knn), k-means clustering, random forest, dimensionality-reducing algorithms, gradient-enhancing algorithms, neural networks (e.g., autoencoder, convolutional, recurrent, perceptrons, long/short-term memory (LSTM ), Hopfield, Boltzmann, Deep Belief, devolutional, generative-adversative, liquid state machine, etc. and/or other types of machine learning models.

Having identified various components in the present disclosure, it should be appreciated that any number of components and arrangements may be employed to achieve desired functionality within the scope of the present disclosure. For example, the components in the embodiments illustrated in the figures are represented with lines for conceptual clarity. However, other arrangements of these and other components may also be implemented. For example, although some components are illustrated as individual components, many of the elements described herein may be implemented as discrete or distributed components, or in cooperation with other components, and in any suitable combination and location. Some elements can also be omitted entirely. Additionally, various functions described herein as being performed by one or more entities may also be performed by hardware, firmware, and/or software, as described below. For example, various functions may be performed by a processor executing instructions stored in memory. As such, other arrangements and elements (e.g., engines, interfaces, functions, orders and groupings of functions, and the like) may be used in addition to or in place of what is shown.

The subject matter of the present invention has been specifically described herein as meeting certain regulatory requirements. However, the description itself is not intended to limit the scope of the present patent. On the contrary, the invention is intended that the claimed subject matter may be embodied in other ways and involve various steps or combinations of steps similar to those described herein, in conjunction with other present or future technologies. Additionally, although the terms "step" and/or "block" may be used herein to denote various elements of the methods employed, the terms should not be construed to imply any particular order among or between various steps disclosed herein, es unless the order of the individual steps is explicitly described.

The present invention has been described in terms of specific embodiments, which are intended in all respects to be illustrative and not restrictive. Alternative embodiments will suggest themselves to one of ordinary skill in the art to which the invention pertains without departing from the scope thereof.

From the foregoing, it is apparent that the present invention is directed to accomplishing all of the purposes and objects set forth above and to providing other advantages that are obvious and inherent in the system and method. It should be appreciated that certain features and sub-combinations are useful and can be used without reference to other features and sub-combinations. This is to be encompassed within and contained within the scope of the claims.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US 17/017344 [0001]

Claims (20)

One or more computer storage media storing computer-usable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations, including: causing a presentation of interactive tiles representing (i) video segments of a segmentation of video and (ii) instances of detected features of the video segments; in one of the interactive tiles, detecting a click or a tap on a visualization of one of the instances of one of the detected features detected from one of the video segments presented by the interactive tile; and Navigating to a portion of the video where the instance of the detected feature is present. One or more computer storage media claim 1 , wherein the interactive tile includes: a thumbnail representing a frame of the video segment, a portion of a transcript of the video segment, a visualization of detected faces in the video segment, and a faceted timeline representing a category of the detected features in the video segment . One or more computer storage media claim 1 or 2 , wherein the interactive tile includes a faceted timeline with facets representing a set of the instances in a category of the detected features in the video segment and corresponding feature areas where the instances in the category are located on the faceted timeline, wherein the clicking or tapping on the visualization occurs on one of the facets on the faceted timeline. One or more computer storage media claim 2 or 3 , wherein the category of detected features includes detected faces, detected categories of sound, detected visual scenes, or detected visual artifacts. One or more computer storage media according to any of Claims 1 until 4 , where the interactive tile is swipeable to show different thumbnails of frames of the video segment. One or more computer storage media according to any of Claims 1 until 5 , the operations further comprising: triggering a re-segmentation of the segmentation of the video to a search segmentation based on a request; and breaking up the interactive tiles to represent smaller units of the video segments identified by the search segmentation. One or more computer storage media according to any of Claims 1 until 6 , where the presentation arranges the interactive tiles in a grid of rows and columns. Computerized or computer-implemented method, comprising: causing a presentation of interactive tiles representing (i) video segments of a segmentation of video and (ii) instances of detected features of the video segments; detecting an interaction with one of the interactive tiles representing one of the video segments; and responsive to detecting the interaction, adding at least a portion of the video segment to a selection of video segments. Computerized procedure claim 8 , wherein detecting the interaction with the interactive tile comprises: detecting a drag operation that drags the interactive tile into a field representing the selection of video segments, or detecting an activation of an interaction element in the interactive tile. Computerized procedure claim 8 or 9 wherein detecting the interaction with the interactive tile comprises: detecting, in the interactive tile, a click or a tap on a visualization of one of the instances of one of the detected features detected from one of the video segments presented by the interactive tile; activating a context menu in response to the click or tap; and detecting an input that selects an option from the context menu to add to the selection the portion of the video segment corresponding to the instance of the detected feature. Computerized method according to one of Claims 8 until 10 , wherein the interactive tile includes: a thumbnail representing a frame of the video segment, a portion of a transcript of the video segment, a visualization of detected faces in the video segment, and a faceted timeline representing a category of the detected features in the video segment . Computerized method according to one of Claims 8 until 11 wherein the interactive tile includes a faceted timeline having facets representing a set of the instances in a category of the detected features in the video segment and corresponding feature regions where the instances in the category are located on the faceted timeline. Computerized method according to one of Claims 8 until 12 , where the interactive tile is swipeable to show different thumbnails of frames of the video segment. Computerized method according to one of Claims 8 until 13 , where the presentation arranges the interactive tiles in a grid of rows and columns. Computer system comprising: one or more hardware processors and a memory configured to provide computer program instructions for the one or more hardware processors; a video interaction engine configured to use the one or more hardware processors to perform operations that include: causing a presentation of a grid of interactive tiles representing video segments of a segmentation of video; detecting an interaction with one of the interactive tiles representing one of the video segments; and in response to detecting the interaction, performing an operation related to the video segment. computer system after claim 15 wherein the interactive tile includes a faceted timeline having facets representing instances in a category of detected features in the video segment and corresponding feature regions where the instances in the category are located on the faceted timeline. computer system after Claim 16 , wherein the category of detected features includes detected faces, detected categories of sound, detected visual scenes, or detected visual artifacts. Computer system according to one of Claims 15 until 17 , where the interactive tile is swipeable to show different thumbnails of frames of the video segment. Computer system according to one of Claims 15 until 18 , wherein detecting the interaction with the interactive tile comprises: detecting a click or a tap on a visualization of an instance of a detected feature that is detected from the video segment, wherein the operation concerning the video segment comprises: navigating to a portion of the video segment, in which the instance of the detected feature is present. Computer system according to one of Claims 15 until 19 , wherein detecting the interaction with the interactive tile comprises: detecting a drag operation that pulls the interactive tile into a field representing a selection of video segments, or detecting an activation of an interaction element in the interactive tile, the operation relating to the video segment includes: adding at least a portion of the video segment to the selection of video segments.

Images