US20180227482A1

US20180227482A1 - Scene-aware selection of filters and effects for visual digital media content

Info

Publication number: US20180227482A1
Application number: US15/427,030
Authority: US
Inventors: Stefan Johannes Josef HOLZER; Matteo Munaro; Abhishek Kar; Alexander Jay Bruen Trevor; Krunal Ketan Chande; Michelle Jung-Ah Ho; Radu Bogdan Rusu
Original assignee: Fyusion Inc
Current assignee: Fyusion Inc
Priority date: 2017-02-07
Filing date: 2017-02-07
Publication date: 2018-08-09
Also published as: US20240267481A1; US20250310468A1; US12381995B2

Abstract

Provided are mechanisms and processes for scene-aware selection of filters and effects for visual digital media content. In one example, a digital media item is analyzed with a processor to identify one or more characteristics associated with the digital media item, where the characteristics include a physical object represented in the digital media item. Based on the identified characteristics, a digital media modification is selected from a plurality of digital media modifications for application to the digital media item. The digital media modification may then be provided for presentation in a user interface for selection by a user.

Description

TECHNICAL FIELD

The present disclosure relates to the selection, recommendation, and application of filters and effects to visual digital media content.

DESCRIPTION OF RELATED ART

Visual digital media content is commonly modified by applying filters and effects. For example, a visual filter may sharpen, blur, or emboss an image to introduce a desired visual effect. However, current techniques are limited in their ability to modify complex digital media content such as video or multi-views. Further, users may face a large number of choices when attempting to select a particular filter or effect to apply to a digital media content item. Accordingly, it is desirable to develop improved mechanisms and processes relating to selecting filters and effects to apply to digital media content items.

Overview

Provided are various mechanisms and processes relating to the selection, recommendation, and application of filters and effects to visual digital media content.
In one aspect, which may include at least a portion of the subject matter of any of the preceding and/or following examples and aspects, a process implemented at a client device and/or embodied in a computer readable media includes analyzing a visual digital media item with a processor to identify one or more characteristics associated with the visual digital media item, where the characteristics include a physical object represented in the visual digital media item. Next, a visual digital media modification is selected from a plurality of visual digital media modifications based on the identified characteristics for application to the visual digital media item. Then, the selected visual digital media modification is provided for presentation in a user interface for selection by a user. The visual digital media item may be a video stream such as a live camera view captured via a camera. In some implementations, the visual digital media item includes a surround view of the object, where the surround view of the object includes spatial information, scale information, and a plurality of different viewpoint images of the object.
In another aspect, which may include at least a portion of the subject matter of any of the preceding and/or following examples and aspects, the characteristics may include structure information indicating a physical context in which the physical object is positioned, pose information indicating an attitude or position associated with the physical object, and/or movement information indicating a degree of velocity or acceleration of the object. The visual digital media modification may include a virtual object positioned within the visual digital media item, an artificial light source that appears to be blocked by the physical object in the visual digital media item, a change to the color of a portion of the visual digital media item, and/or motion blur indicating movement associated with the object. The object may be a human being, and the visual digital media modification may include a text bubble appearing in proximity to a face.
In yet another aspect, which may include at least a portion of the subject matter of any of the preceding and/or following examples and aspects, analyzing the visual digital media item may involve receiving the visual digital media item at a server via a network from a client device, where providing the visual digital media modification for presentation in a user interface includes transmitting a message via the network to the client device. In still another aspect, analyzing the visual digital media item may involve transmitting the visual digital media item to a server via a network from a client device and receiving a response message at the client device, the response message identifying the one or more characteristics.
These and other embodiments are described further below with reference to the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings, which illustrate particular embodiments of the present invention.

FIG. 1 illustrates one example of a process for performing visual digital media modification selection.

FIG. 2 illustrates one example of a process for performing visual digital media modification preprocessing.

FIG. 3 illustrates one example of a process for performing pose detection for an object.

FIG. 4 illustrates one example of a system that can be used to perform live video stream filtering.

FIG. 5 illustrates one example of a process for performing live filtering of a video stream.

FIG. 6 illustrates one example of a process for performing live filter processing of a video stream at a server.

FIG. 7 illustrates an example of a surround view acquisition system.

FIG. 8 illustrates an example of a device capturing multiple views of an object of interest.

FIG. 9 illustrates an example of a device capturing views of an object of interest to create a multi-view media representation to which a filter may be applied.

FIG. 10 illustrates a particular example of a computer system that can be used with various embodiments of the present invention.

DETAILED DESCRIPTION

Reference will now be made in detail to some specific examples of the present disclosure including the best modes contemplated by the inventors for carrying out the present disclosure. Examples of these specific embodiments are illustrated in the accompanying drawings. While the present disclosure is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the present disclosure to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the present disclosure as defined by the appended claims.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Particular example embodiments of the present invention may be implemented without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
Various techniques and mechanisms of the present invention will sometimes be described in singular form for clarity. However, it should be noted that some embodiments include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise. For example, a system uses a processor in a variety of contexts. However, it will be appreciated that a system can use multiple processors while remaining within the scope of the present invention unless otherwise noted. Furthermore, the techniques and mechanisms of the present invention will sometimes describe a connection between two entities. It should be noted that a connection between two entities does not necessarily mean a direct, unimpeded connection, as a variety of other entities may reside between the two entities. For example, a processor may be connected to memory, but it will be appreciated that a variety of bridges and controllers may reside between the processor and memory. Consequently, a connection does not necessarily mean a direct, unimpeded connection unless otherwise noted.
According to various embodiments, improved mechanisms and processes are described for the selection, application, and recommendation of modifications to visual digital media items. Such modifications may include image filters, virtual objects, digital effects, or other such alterations. Such improved mechanisms and process allow a user to be presented with modifications that are specifically relevant to the content, structure, context, motion, and/or poses of a visual digital media item. In this way, a user may be presented with only the relevant modifications and need not manually sift through a large number of irrelevant modifications. Further, if modifications are applied in a way that reflects the modified visual digital media item, then the modifications may appear to be more realistic, and a larger number of modifications may be possible. Also, specific modifications may be applied to different parts of the same scene.
Digital media modifications, also referred to herein as filters, modify and/or add to the visual data of a visual digital media item such as a static image, a video stream, or a multi-view interactive digital media representation. According to various embodiments, filters can include any techniques for altering a visual digital media item. For example, a filter can alter the color information of the captured visual data by changing contrast, changing brightness, or applying a transformation to the underlying color matrix. As another example, a filter can add additional elements to the scene such as 2D or 3D stickers or text placed relative to an object in the visual digital media item, or any other such alteration. An artificial object can be placed relative to a two-dimensional object in two-dimensional space and/or relative to the three-dimensional reference coordinate system of a multi-view interactive digital media representation in three-dimensional space. The application of the filters can happen live (e.g., in a live media stream or a in a camera view) or in post-processing.
According to various embodiments, techniques and mechanisms described herein provide for improved user experience of a computer system. In many contexts, large set of different filters can be applied to visual digital media content. Due to the vast number of different filters that can be applied, conventional techniques require either a user to manually review a large number of filters to find the best filter for a certain use case, or application developers to limit the amount of filters the user can select from. Both options are suboptimal for the user. In contrast, techniques and mechanisms described herein allow a user to be presented with filters specifically relevant to the user's content.
With reference FIG. 1, shown is one example of a process for performing visual digital media modification selection. In some implementations, the process 100 may be performed at a client device. Alternately, the process 100 may be performed at a server. In yet another implementation, some operations shown in the process 100 may be performed at client device while other operations are performed at a server in communication with the client device. In particular embodiments, the client device and the server may be implemented as different processes running on the same physical device. In other embodiments, the client device and the server may be implemented on different physical devices in communication via a network.
According to various embodiments, the process 100 begins when a request to apply a filter to a visual digital media item is received at 102. The request may be received when a user specifically requests to apply a filter to a visual digital media item. Alternately, the request may be generated automatically when triggered by a particular action such as the activation of a camera at a client device.
Next, at 104, the visual digital media item is analyzed to identify content and structure information. According to various embodiments, content information may include any indication of objects represented in the visual digital media item. For instance, the identified content may include a human being, an animal, a plant, text, an inanimate object such as a vehicle, or an abstract shape such as a ball. Structure information may include a ground plane or a wall. Semantic areas such as sky, grass, and water may also be identified.
Then, at 106, the visual digital media item is analyzed to identify pose and movement information. In some implementations, pose information may indicate an attitude or position of an identified object. For example, if a human being is identified as being represented in the visual digital media item, then pose information may indicate whether the human being is sitting, standing, walking, or arranged in some other posture. Pose information may also be applied to other types of objects. For instance, pose information may indicate the position of a vehicle relative to the viewer, the stance of an animal, the attitude or position of a deformable inanimate object, or other such positioning information. Movement information may indicate the velocity or acceleration of an identified object. The movement may be identified relative to scene structure, another object, the viewpoint, or any other reference plane or point.
A variety of techniques may be used to identify content, structure, pose, and movement information. Identifying such information may involve applying a content recognition algorithm to visual media. For instance, a recognition algorithm may be applied to an image, a video frame, one or more images in a multi-view of an object, or a stream of video frames. The specific techniques used to identify content, structure, pose, and movement information may depend on the particular implementation. For instance, different techniques may be used based on characteristics of the visual digital media item or the identify content, structure, pose, or movement information to be identified.
After analyzing the visual digital media item, one or more visual digital media modifications to present are selected at 108. According to various embodiments, one or more specific filters may be selected based on the content and/or context of the scene depicted in the visual digital media content. That is, the identification of objects such as cars, food or people, or the identification of semantic areas such as sky, grass, water may trigger the selection, recommendation, and/or application of specific filters. For example, if a car is present in the captured scene and is detected, then car-specific filters such as stickers, exhaust fumes, and other types of augmented reality modifications may be made available. As another example, if a person is present in the scene then filters specific for humans may be added, such as speech bubbles. As still another example, filters may be selected that apply to particular contexts, such as fashion shots or sporting events. For some objects, such as people and vehicles, poses and movements may also be used to trigger filters. If multiple objects are present, such as both a person and a vehicle, then filters may be selected that are specific to the combination of objects, instead of or in addition to filters that are specific to only persons or only vehicles.
In particular embodiments, selected filters may be applied automatically to the visual digital media modification. For example, the server may transmit an instruction to the client machine to apply one or more selected filters. Alternately, or additionally, one or more filters may be provided for manual selection by a user in a user interface. For instance, the server may transmit an instruction to the client machine identifying one or more filters to present.
In some implementations, one or more of a variety of context-specific filters may be selected. In one example, certain filters may be selected for certain objects. For instance, vehicles may be modified to include stickers, color changes, motion blur, or other such vehicle-specific alterations. In a second example, moving artificial objects may be added that react with the scene structure such as balls that bounce off of the ground, objects that accumulate on the ground, weather patterns such as rain or snow that interact with the scenery, artificial light sources that are blocked by objects in the scene, or other such alterations. In a third example, an artificial object may be automatically positioned relative to the scene and/or objects. Specific examples of inserting virtual objects include, but are not limited to: 3D text bubbles appearing next to a human's face, “vroom” text appearing near a car's engine, or hats or clothing being placed on humans or animals. In a fourth example, a filter may change the color of the sky or alter the “style” of a visual digital media item. For instance, the style may be altered to appear cartoonish or retro.
In some embodiments, effects and filters may be added based on a detected pose. In one example, if a person is detected in a particular pose, then a filter may be selected that will cause a laser beam or fire to shoot from the person's hands. In a second example, if multiple people are detected close to one another, then effects or elements which correlate both people can be added, such as hearts indicating affection. In a third example, if a person is detected, then the person can be cut out of the scene and pasted in one or more times in a different position or pose, such as dancing. In a fourth example, parts of a human body can be replaced. For instance, a head may be replaced with an apple, a crocodile head, or some other object. In a fifth example, the context can be changed. For instance, when a person is detected as jumping, a filter may be applied to depict alligators or some other hazard beneath the person. In a sixth example, elements can be added that interact with a person. For instance, a snake can be added that moves up a person's body, or lightning can be added that traverses a person's body. An infinite variety of such alterations is possible. In a seventh example, a person may be cut or copied from a scene and then pasted back into the same scene one or more times in the same pose but a different position. For instance, a single dancing person may be copied multiple times in the same pose to create a crowd of dancing people.
In particular embodiments, analyzing the visual digital media item may involve transmitting some or all of the visual digital media item to a server for processing. The server may then process the visual digital media item and response with one or more recommend digital media modifications. Techniques for client-server interactions are discussed in greater detail with respect to FIGS. 4, 5, and 6.
According to various embodiments, visual digital media modifications may be selected based at least in part on explicit classifications or categorizations. For instance, a user or system administrator may specifically identify a particular type of modification as pertaining to a vehicle, a person, or both. Alternately, or additionally, visual digital media modifications may be selected based at least in part on implicit or machine-generated classifications or categorizations. For instance, the system may analyze user selections to help determine which modifications to suggest for which visual digital media items. Additional details regarding machine classification are discussed with respect to FIG. 2.
Once the one or more visual digital media modifications are selected, at 110 the selections are provided for presentation in a user interface. According to various embodiments, the specific technique for providing the selections for presentation may depend in part on the specific implementation. For instance, if the analysis is performed at a server, then providing the selections for presentation may involve transmitting a message to a client device with instructions for presenting the selections in a user interface at the client device. Alternately, if the analysis is performed at a client device, then the selections may be presented directly in a user interface.
Then, a determination is made at 112 as to whether to continue to analyze the visual digital media item. According to various embodiments, analysis may continue until one or more conditions or criteria are met. These may include, but are not limited to: the receipt of user input indicating a request to stop analysis, the selection of a particular filter or filters for presentation, and the termination of a live or prerecorded media stream.
With reference FIG. 2, shown is one example of a process for performing visual digital media modification preprocessing. According to various embodiments, the procedure 200 may be performed in order to help determine which visual digital media modifications to suggest for which visual digital media items. To accomplish this, user-selections of filters may be analyzed to identify characteristics likely to make particular filters attractive in specific situations. The procedure 200 may be performed at a server having access to a wide range of user selections or at a client machine. The procedure 200 may be initiated when a request is received at 202 to perform visual digital media modification preprocessing. According to various embodiments, the request may be generated automatically (for instance periodically) or manually (for instance by a system administrator).
Next, a visual digital media item is selected for analysis at 204. In some implementations, a visual digital media item may be selected for analysis when it is associated with user-provided input indicating a selection of a filter for application to the item. In one example, all filtered visual digital media items may be selected for analysis in some sequence. Alternately, a subset of the available data may be analyzed. For instance, a designated number of filtered visual digital media items may be selected in a particular category, such as filtered visual digital media items that include people, vehicles, or animals. Once a visual digital media item is selected for analysis, it is analyzed at 206 to identify content and structure information and at 208 to identify pose and movement information. These analyses may be substantially similar to the operations 104 and 106 discussed with respect to FIG. 1.
After the visual digital media item is analyzed, one or more user-selected visual digital media modification is identified at 210. For instance, a user initially viewing the media item may have been presented with a set of filters to choose from and then selected a particular filter to apply to the media item. The user's choice may then have been recorded for further analysis. At 212, one or more weights associated with the identified content, structure, pose, and/or movement information is updated.
According to various embodiments, updating the one or more weights may involve indicating a connection between a characteristic of the media item and the selected filter. In one example, a particular filter is identified as having been applied to a particular visual digital media item that analysis reveals to include a moving car, stationary flowers, and a walking dog. In this example, the weights linking the particular filter to each of these characteristics are increased. For instance, if the filter includes flames that shoot in a particular direction, then users may be more likely to apply the filter to moving cars. Over time, the weight linking the filter to the media item characteristic of a moving car may be increased so that eventually the system may automatically recommend the shooting flames filter when the media item includes a car without needing a human to explicitly flag the filter as being specific to vehicles.
Next, a determination is made at 214 as to whether to continue to analyze visual digital media items. According to various embodiments, analysis may continue until one or more conditions or criteria are met. These may include, but are not limited to: the receipt of user input indicating a request to stop analysis, the selection of a particular set of weights, and the analysis of all available user selections.
FIG. 3 illustrates one example of a process for performing pose detection for an object. According to various embodiments, pose detection may be used for any of various purposes. In one example, pose detection may be used to trigger a filter. For instance, pose detection may be used to determine if a person is exhibiting a particular pose, such as pumping a fist in the air. If such a pose is detected, then the video stream may be altered to depict lightning extending from the fist. In another example, skeleton detection may be used to trigger a photo to be captured. For instance, a person may position a camera to take a self-image and then move in front of the camera. The camera may then capture an image when it identifies the person's skeleton and determines that the person has stopped moving or has entered into a particular pose, such as jumping in the air. The procedure 300 is initiated when a request is received at 302 to perform pose detection for a video stream.
Then, at 304, prior video frame information associated with the video stream is identified. At 306 skeleton detection operations for the video stream are performed. According to various embodiments, skeleton detection operations may be performed using any of various suitable methods. In one example, a convolutional neural network may be applied to an image to first detect all objects in the scene and then estimate the skeleton joints for those that belong to the “person” category. In a second example, static skeleton detection at the server may be combined with server-side skeleton detection and/or tracking across prior frames. For instance, the results of one or more skeleton detection operations for previous skeleton detection messages may be analyzed to aid in the detection of a skeleton for the current frame. In a third example, non-visual data such as accelerometer or gyroscopic data may be analyzed to aid in skeleton detection.
After performing skeleton detection, at 308 pose detection is performed. In pose detection, the detected human skeleton may be used to determine whether the arrangement of the skeleton at a particular point in time matches one or more of a discrete set of human poses. In some implementations, pose detection may be accomplished by first estimating a homography from the skeleton joints that in order to frontalize the skeleton for a bitter pose. Then, pose detection may be performed by analyzing spatial relations of the frontalized joints. Next, a temporal filtering method may be applied to remove spurious detections. In particular embodiments, such techniques may be applied to detect poses for either individuals or for multiple people.
In some embodiments, pose detection may involves scaling or stretching location information associated with the detected skeleton and then comparing the location information with the predetermined location information associated with specific poses, where a high degree of similarity between the detected skeleton information and the predetermined skeleton pose information would indicate a match. When pose detection is used, different poses may trigger different events, such as the application of specific filters to a video stream. Alternately, or additionally, the detection of a specific pose may trigger the recommendation of one or more filters to the user for the user to select. In either case, pose detection may be used to suggest or identify start and/or end times for an effect as well as the type of effect that could be added.
With reference to FIG. 4, shown is one example of a system that can be used to perform a live video stream filtering. As depicted, a combination of client and server applications is used to implement a filtering mechanism that runs live in a capture device application, such as with a camera on a smartphone. While the camera is recording, the user points the camera at an object. The smartphone then communicates with the server, and collectively the two devices analyze the video stream to provide a filtered view of the video stream in real time.
In the present embodiment, the client is depicted as device 404, which can be a capture device such as a digital camera, smartphone, mobile device, etc. The server is depicted as system 402, which receives images selected from the video stream at the client device. The video stream at the client device is divided into video frames 451 through 461. The server processes the frames sent from the client device and response with filtering information that can be used to apply a filter to the video stream at the client device. The client device includes a camera 406 for capturing a video stream, a communications interface 408 capable of communicating with the server, a processor 400, memory 402, and a display screen 404 on which the video stream may be presented.
According to various embodiments, the client and server may coordinate to apply a filter to the video stream at least in part due to limited computing resources at the client machine. However, as discussed herein, the network latency and processing time involved in transmitting video frames to the server means that the video stream at the client device has progressed to a new video frame before receiving the filter processing message from the server with the filter information associated with the preceding frame. For instance, in FIG. 4, the first request 471 transmits the frame 451 to the server, while the first response 472 corresponding to the frame 451 arrives while the frame 455 is being processed. Similarly, the second request 473 and third request 474 transmit frames 455 and 457 respectively, but the corresponding second and third responses 475 and 476 are not received until the video stream has arrived at frames 459 and 461 respectively.
In some implementations, the client application sends (and also receives) data in a sparse manner, meaning that data is sent to the server potentially not for all frames captured by the camera. Therefore, in order to present a filtered result for a live stream, the information received from the server is tracked or propagated to new frames received from the camera until new information from the server is available. For example, in FIG. 4, the client device may propagate information received in the first response 472 through frames 456, 457, and 458 until the second response 475 is received for the processing of frame 459.
With reference to FIG. 5, shown is one example of a process for performing live filtering of a video stream. According to various embodiments, the process shown in FIG. 5 may be performed at a client machine in communication with a server, such as the client machine 104 in communication with the server 105 shown in FIG. 1. The two devices may coordinate to split the processing operations required to apply a filter to a live video stream.
In the present example, a live filtering process 500 begins with the client device receiving a request to perform filtering of a video stream at 502. In some implementations, the request may be generated based on user input requesting the application of a filter. Alternately, the request may be generated automatically when the client device detects that a video stream is being captured or displayed at the client device. Next, the system select a video stream frame for processing at 504. According to various embodiments, video stream frames may be processed sequentially. For instance, each frame in a live video stream may be processed prior to presenting the video stream to the user so that a filter may be applied.
At 506, a determination is made as to whether the selected video stream frame meets a designated criterion. In some implementations, any of a variety of criteria may be used to select a video stream frame for transmission to the server. For example, if the filtering process has just been initiated, then the client device may select the first available video stream frame for processing. As another example, one or more criteria may be applied to select the video stream frame. For instance, the client device may select a video stream frame that exceeds a threshold level of light or detail to allow for sufficient information for applying a filter. As yet another example, the client device may select a video stream frame for processing after a designated period of time or number of frames have passed since the last video stream frame was transmitted to the server.
If the selected frame meets the designated criterion, then information about the selected frame is transmitted to the server at 508. According to various embodiments, a variety of information may be transmitted to the server. In one example, some or all of the image data associated with the frame may be transmitted. For instance, the entire frame may be transmitted. Alternately, the frame may be compressed or down sampled to reduce bandwidth usage. In a second example, IMU information such as gyroscopic data, compass data, or accelerometer data may be transmitted. This IMU information may provide data about the position, velocity, acceleration, direction, rotation, or other such characteristics of the device around the time that the frame was captured. In a third example, GPS information may be transmitted. In some implementations, the specific information transmitted to the server may depend on the type of processing being performed at the server and/or the type of filter being applied at the client device.
Next, a determination is made at 510 as to whether a new filter processing message has been received from the server. As shown in FIG. 1, the server sends messages that include information for applying filters to frames, but these filter processing messages are sent at a lag when compared with the live processing and presentation of the video stream.
If no new filter processing message has been received, then at 512 a filter is applied based on existing data that is locally available at the client machine. In some embodiments, applying a filter based on locally available data may involve propagating information from one frame to another. For instance, a current frame may be analyzed to identify the same feature (e.g., an object corner or an area of color) that was identified in the preceding frame. According to various embodiments, a multitude of approaches can be used to propagate information from one frame to another. One such approach is frame-to-frame tracking, which can be based on information that may include, but is not limited to: tracking of sparse keypoints, dense or sparse optical flow, patch tracking, tracking of geometric instances, or other such information. Another such approach is frame-to-frame matching, which involve techniques that may include, but are not limited to: descriptor based matching of keypoints which are detected in both frames, patch matching, detection and matching of higher level features (e.g. a human face), or other such techniques. Both approaches can focus the tracking and matching efforts on regions or features of interest if such regions or features are identified.
In some implementations, a special processing cases the time from the first frame that is sent to the server to the frame when the corresponding results are received back from the server. Since there is no server-created scene interpretation available until the results of the first frame are received, the client device may not know which specific information in the scene needs to be propagated. Various approaches are possible for handling this situation. In one example, all or most information in the frame is equally propagated. For instance, keypoints may be distributed over the whole image. In a second example, an efficient method for estimating one or more regions of interest may be applied on the client device. For instance, a bounding box for the region may be computed. Then, the propagation of information may be concentrated on the region or regions of interest. In a third example, matching methods may be applied to directly match the information extracted from the first frame to the frame after which the results from the server are available.
If instead a new filter processing message has been received, then at 514 a filter is applied based on both the locally available data and the data provided by the server. According to various embodiments, new information received from the server may be combined with the information propagated from frame to frame. To accomplish this goal, various approaches may be used. In one example, old information may be replaced with new information received from the server. In a second example, old information may be combined with new information in a weighted fashion, for instance based on relative confidence values associated with server results and propagation results.
According to various embodiments, the specific operations performed to apply a filter may depend in large part upon the specific type of filter being applied. In one example, a caption bubble may be applied to a video of a person when the person exhibits a particular pose. In this first example, the server may perform skeleton detection to facilitate pose estimation while the client device tracks low-level image features such as a point associated with a person's elbow or a surface area that is part of the background. Then, the client device may combine the low-level feature tracking information with the skeleton detection information provided by the server to determine whether the person is positioned in the particular pose. In a second example, a filter may be applied to a vehicle based on its position (e.g., crossing a finish line). In this second example, the server may perform segmentation to identify the segmentation and characteristics of the vehicle, while the client device tracks low-level features such as shapes to propagate the location of the vehicle between communication with the server.
After applying the filter to the selected frame, the filtered frame is provided for presentation at 516. In some implementations, providing the filtered frame for presentation may involve displaying the filtered frame as part of the video stream on a display screen. Alternately, or additionally, the filtered frame may be stored to memory and or persistent storage for later playback. In a different example, the filtered frame may be transmitted to a separate device for presentation, such as an augmented reality or virtual reality device in communication with the client device. Finally, at 518 a determination is made as to whether to process an addition frame. According to various embodiments, additional frames may be processed until any of a variety of conditions are met. These conditions may include, but are not limited to: receiving user input indicating a request to terminate live filtering, determining that the video stream has terminated, or determining that the server is inaccessible via the network.
With reference to FIG. 6, shown is one example of a configuration for performing live filtering server processing. In some implementations, the procedure 600 may be performed in order to perform server-side processing to facilitate the live filtering of a media stream at a client device. The procedure 600 may be initiated at 602 when a live filtering request message for a video stream is received from a client device. According to various embodiments, as discussed with respect to FIGS. 1 and 2, a variety of information may be included in a live filtering request message. For instance, the request message may include the identity of the client device as well as any information necessary for performing live filtering, such as image data information associated with one or more video stream frames, IMU information, or GPS information.
In particular embodiments, after receiving the request message, at 604 the server may identify information associated with one or more prior video frames in the video stream. For example, the prior video frame information may include any raw data transmitted from the client device in early live filtering request messages. Alternately, or additionally, the prior video frame information may include processed or filtered data generated by processing previous live filtering request messages.
Then, at 606 the server performs filter processing operations for the video stream. The specific filter processing operations performed may depend in large part on the particular type of filter being applied to the video stream. Some examples of processing operations running on the server may include, but are not limited to: detection, segmentation, and pose estimation. Such methods may be applied to objects that include, but are not limited to: humans, animals, vehicles, inanimate objects, and plants. Other examples of methods running on the server may include, but are not limited to: depth estimation, scene reconstruction, scene decomposition, and semantic labeling.
According to various embodiments, the filter processing operations may generate a wide range of information. For example, the filter processing operations may generate location information that identifies locations of high-level features such as faces or skeleton components on image data sent from the client device. As another example, the filter processing operations may include or identify virtual elements to overlay on top of the video stream at the client device. For instance, the filter processing operation may identify the video stream as including footage of a running dog and then indicate as one filter possibility a cape that could be overlain on the moving image of the dog to generate a visual effect of a “super dog.” As yet another example, the filter processing operations may include semantic elements such as labels for recognized objects or words generated by applying optical character recognition to text in the video stream.
After performing the filter processing operations, at 608 a filter processing message is transmitted to the client device. According to various embodiments, the filter processing message may include any suitable information, including location information identifying the location of features in the scene, semantic information that identifies the meaning of particular elements in the scene, or data for overlaying virtual elements on top of the scene.
With reference to FIG. 7, shown is an example of a surround view acquisition system that can be used to generate a multi-view interactive digital media representation that can be used for the application of filters or visual effects. A multi-view interactive digital media representation includes much more information than a single image. Whereas a single image may include information such as a grid of color pixels and the date/time of capture, a multi-view interactive digital media representation includes information such as such as grids of color pixels, date/time of capture, spatial information (flow/3D), location, and inertial measurement unit information (IMU) (i.e., compass, gravity, orientation). A multi-view interactive digital media representation brings focus to an object of interest because it provides separation between the foreground and background. In addition, a multi-view interactive digital media representation provides more information about the scale, context, and shape of the object of interest. Furthermore, by providing multiple views, aspects of the object that are not visible from a single view can be provided in a multi-view interactive digital media representation.
In the present example embodiment, the surround view acquisition system 700 is depicted in a flow sequence that can be used to generate a surround view. According to various embodiments, the data used to generate a surround view can come from a variety of sources. In particular, data such as, but not limited to two-dimensional (2D) images 704 can be used to generate a surround view. These 2D images can include color image data streams such as multiple image sequences, video data, etc., or multiple images in any of various formats for images, depending on the application. Another source of data that can be used to generate a surround view includes location information 706. This location information 706 can be obtained from sources such as accelerometers, gyroscopes, magnetometers, GPS, Wi-Fi, IMU-like systems (Inertial Measurement Unit systems), and the like. Yet another source of data that can be used to generate a surround view can include depth images 708. These depth images can include depth, 3D, or disparity image data streams, and the like, and can be captured by devices such as, but not limited to, stereo cameras, time-of-flight cameras, three-dimensional cameras, and the like.
In the present example embodiment, the data can then be fused together at sensor fusion block 710. In some embodiments, a surround view can be generated a combination of data that includes both 2D images 704 and location information 706, without any depth images 708 provided. In other embodiments, depth images 708 and location information 706 can be used together at sensor fusion block 710. Various combinations of image data can be used with location information at 706, depending on the application and available data.
In the present example embodiment, the data that has been fused together at sensor fusion block 710 is then used for content modeling 711 and context modeling 714. During this process, the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, although the content can be a two-dimensional image in some embodiments. Furthermore, in some embodiments, the context can be a two-dimensional model depicting the scenery surrounding the object of interest. Although in many examples the context can provide two-dimensional views of the scenery surrounding the object of interest, the context can also include three-dimensional aspects in some embodiments. For instance, the context can be depicted as a “flat” image along a cylindrical “canvas,” such that the “flat” image appears on the surface of a cylinder. In addition, some examples may include three-dimensional context models, such as when some objects are identified in the surrounding scenery as three-dimensional objects. According to various embodiments, the models provided by content modeling 711 and context modeling 714 can be generated by combining the image and location information data.
According to various embodiments, context and content of a surround view are determined based on a specified object of interest. In some examples, an object of interest is automatically chosen based on processing of the image and location information data. For instance, if a dominant object is detected in a series of images, this object can be selected as the content. In other examples, a user specified target 702 can be chosen. It should be noted, however, that a surround view can be generated without a user specified target in some applications.
In the present example embodiment, one or more enhancement algorithms can be applied at enhancement algorithm(s) block 716. In particular example embodiments, various algorithms can be employed during capture of surround view data, regardless of the type of capture mode employed. These algorithms can be used to enhance the user experience. For instance, automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used during capture of surround view data. In some examples, these enhancement algorithms can be applied to image data after acquisition of the data. In other examples, these enhancement algorithms can be applied to image data during capture of surround view data.
According to particular example embodiments, automatic frame selection can be used to create a more enjoyable surround view. Specifically, frames are automatically selected so that the transition between them will be smoother or more even. This automatic frame selection can incorporate blur- and overexposure- detection in some applications, as well as more uniformly sampling poses such that they are more evenly distributed.
In some example embodiments, stabilization can be used for a surround view in a manner similar to that used for video. In particular, key frames in a surround view can be stabilized to produce improvements such as smoother transitions, improved/enhanced focus on the content, etc. However, unlike video, there are many additional sources of stabilization for a surround view, such as by using IMU information, depth information, computer vision techniques, direct selection of an area to be stabilized, face detection, and the like.
For instance, IMU information can be very helpful for stabilization. In particular, IMU information provides an estimate, although sometimes a rough or noisy estimate, of the camera tremor that may occur during image capture. This estimate can be used to remove, cancel, and/or reduce the effects of such camera tremor.
In some examples, depth information, if available, can be used to provide stabilization for a surround view. Because points of interest in a surround view are three-dimensional, rather than two-dimensional, these points of interest are more constrained and tracking/matching of these points is simplified as the search space reduces. Furthermore, descriptors for points of interest can use both color and depth information and therefore, become more discriminative. In addition, automatic or semi-automatic content selection can be easier to provide with depth information. For instance, when a user selects a particular pixel of an image, this selection can be expanded to fill the entire surface that touches it. Furthermore, content can also be selected automatically by using a foreground/background differentiation based on depth. In various examples, the content can stay relatively stable/visible even when the context changes.
According to various examples, computer vision techniques can also be used to provide stabilization for surround views. For instance, key points can be detected and tracked. However, in certain scenes, such as a dynamic scene or static scene with parallax, no simple warp exists that can stabilize everything. Consequently, there is a trade-off in which certain aspects of the scene receive more attention to stabilization and other aspects of the scene receive less attention. Because a surround view is often focused on a particular object of interest, a surround view can be content-weighted so that the object of interest is maximally stabilized in some examples.
Another way to improve stabilization in a surround view includes direct selection of a region of a screen. For instance, if a user taps to focus on a region of a screen, then records a convex surround view, the area that was tapped can be maximally stabilized. This allows stabilization algorithms to be focused on a particular area or object of interest.
In some examples, face detection can be used to provide stabilization. For instance, when recording with a front-facing camera, it is often likely that the user is the object of interest in the scene. Thus, face detection can be used to weight stabilization about that region. When face detection is precise enough, facial features themselves (such as eyes, nose, mouth) can be used as areas to stabilize, rather than using generic key points.
According to various examples, view interpolation can be used to improve the viewing experience. In particular, to avoid sudden “jumps” between stabilized frames, synthetic, intermediate views can be rendered on the fly. This can be informed by content-weighted key point tracks and IMU information as described above, as well as by denser pixel-to-pixel matches. If depth information is available, fewer artifacts resulting from mismatched pixels may occur, thereby simplifying the process. As described above, view interpolation can be applied during capture of a surround view in some embodiments. In other embodiments, view interpolation can be applied during surround view generation.
In some examples, filters can also be used during capture or generation of a surround view to enhance the viewing experience. Just as many popular photo sharing services provide aesthetic filters that can be applied to static, two-dimensional images, aesthetic filters can similarly be applied to surround images. However, because a surround view representation is more expressive than a two-dimensional image, and three-dimensional information is available in a surround view, these filters can be extended to include effects that are ill-defined in two dimensional photos. For instance, in a surround view, motion blur can be added to the background (i.e. context) while the content remains crisp. In another example, a drop-shadow can be added to the object of interest in a surround view.
In various examples, compression can also be used as an enhancement algorithm 716. In particular, compression can be used to enhance user-experience by reducing data upload and download costs. Because surround views use spatial information, far less data can be sent for a surround view than a typical video, while maintaining desired qualities of the surround view. Specifically, the IMU, key point tracks, and user input, combined with the view interpolation described above, can all reduce the amount of data that must be transferred to and from a device during upload or download of a surround view. For instance, if an object of interest can be properly identified, a variable compression style can be chosen for the content and context. This variable compression style can include lower quality resolution for background information (i.e. context) and higher quality resolution for foreground information (i.e. content) in some examples. In such examples, the amount of data transmitted can be reduced by sacrificing some of the context quality, while maintaining a desired level of quality for the content.
In the present embodiment, a surround view 718 is generated after any enhancement algorithms are applied. The surround view can provide a multi-view interactive digital media representation. In various examples, the surround view can include three-dimensional model of the content and a two-dimensional model of the context. However, in some examples, the context can represent a “flat” view of the scenery or background as projected along a surface, such as a cylindrical or other-shaped surface, such that the context is not purely two-dimensional. In yet other examples, the context can include three-dimensional aspects.
According to various embodiments, surround views provide numerous advantages over traditional two-dimensional images or videos. Some of these advantages include: the ability to cope with moving scenery, a moving acquisition device, or both; the ability to model parts of the scene in three-dimensions; the ability to remove unnecessary, redundant information and reduce the memory footprint of the output dataset; the ability to distinguish between content and context; the ability to use the distinction between content and context for improvements in the user-experience; the ability to use the distinction between content and context for improvements in memory footprint (an example would be high quality compression of content and low quality compression of context); the ability to associate special feature descriptors with surround views that allow the surround views to be indexed with a high degree of efficiency and accuracy; and the ability of the user to interact and change the viewpoint of the surround view. In particular example embodiments, the characteristics described above can be incorporated natively in the surround view representation, and provide the capability for use in various applications. For instance, surround views can be used in applying filters or visual effects.
According to various example embodiments, once a surround view 718 is generated, user feedback for acquisition 720 of additional image data can be provided. In particular, if a surround view is determined to need additional views to provide a more accurate model of the content or context, a user may be prompted to provide additional views. Once these additional views are received by the surround view acquisition system 700, these additional views can be processed by the system 700 and incorporated into the surround view.
With reference to FIG. 8, shown is an example of a device capturing multiple views of an object of interest from different locations. The capture device is indicated as camera 811, and moves from location 822 to location 824 and from location 824 to location 826. The multiple camera views 802, 804, and 806 captured by camera 811 can be fused together into a three-dimensional (3D) model. According to various embodiments, multiple images can be captured from various viewpoints and fused together to provide a multi-view digital media representation.
In the present example embodiment, camera 811 moves to locations 822, 824, and 826, respectively, along paths 828 and 830, in proximity to an object of interest 808. Scenery can surround the object of interest 808 such as object 88. Views 802, 804, and 806 are captured by camera 811 from locations 822, 824, and 826 and include overlapping subject matter. Specifically, each view 802, 804, and 806 includes the object of interest 808 and varying degrees of visibility of the scenery surrounding the object 810. For instance, view 802 includes a view of the object of interest 808 in front of the cylinder that is part of the scenery surrounding the object 808. View 804 shows the object of interest 808 to one side of the cylinder, and view 806 shows the object of interest without any view of the cylinder.
In the present example embodiment, the various views 802, 804, and 806 along with their associated locations 822, 824, and 826, respectively, provide a rich source of information about object of interest 808 and the surrounding context that can be used to produce a multi-view digital media representation, such as a surround view. For instance, when analyzed together, the various views 802, 804, and 806 provide information about different sides of the object of interest and the relationship between the object of interest and the scenery. These views also provide information about the relative size and scale of the object of interest in relation to the scenery. Furthermore, views from different sides of the object provide information about the shape and texture of the object. According to various embodiments, this information can be used to parse out the object of interest 808 into content and the scenery 88 as the context. In particular examples, the content can then be used for applying filters.
With reference to FIG. 9, shown is an example of a device capturing views of an object of interest. During a filter session, multiple views of the object 908 may be captured by the device 970 from different locations. In the present example, data is acquired when a user taps a record button 980 on capture device 970 to begin recording images of the object.
The user moves 828 the capture device 970 from location 822 to location 824 along path 828 and from location 824 to location 826 along path 830. As described in more detail throughout this application, filtering can be provided at the device 970, and prompts for the user to capture particular views can be provided during the session. In particular, the system can prompt the user to move the device 970 in a particular direction or may prompt the user to provide additional information. As the user records different views of the object, filtering suggestions may be reiteratively refined to provide accurate results. The user may choose to stop recording by tapping the record button 980 again. In other examples, the user can tap and hold the record button during the session, and release to stop recording. In the present embodiment, the recording captures a series of images that can be used to generate a multi-view digital media representation that can be for filtering either in real-time or after-the-fact.
In some implementations, applying a filter to a multi-view digital media representation may involve processing a succession of images taken from different perspectives. In such an example, the client device may perform low-level processing such as two-dimensional analysis of individual images. The server, on the other hand, may perform high-level processing such as combining different individual images to produce a three-dimensional model of an object that is the subject of a multi-view video.
With reference to FIG. 10, shown is a particular example of a computer system that can be used to implement particular examples of the present invention. For instance, the computer system 1000 can be used to apply one or more filters or effects to visual digital media content according to various embodiments described above. According to particular example embodiments, a system 1000 suitable for implementing particular embodiments of the present invention includes a processor 1001, a memory 1003, a communications interface 1011, a filter interface 1013, and a bus 1015 (e.g., a PCI bus). The filter interface 1013 may include separate input and output interfaces, or may be a unified interface supporting both operations. When acting under the control of appropriate software or firmware, the processor 1001 is responsible for such tasks such as optimization. Various specially configured devices can also be used in place of a processor 1001 or in addition to processor 1001. The complete implementation can also be done in custom hardware. The communications interface 1011 is typically configured to send and receive data packets or data segments over a network. Particular examples of interfaces the device supports include Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like.
In addition, various very high-speed interfaces may be provided such as fast Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces and the like. Generally, these interfaces may include ports appropriate for communication with the appropriate media. In some cases, they may also include an independent processor and, in some instances, volatile RAM. The independent processors may control such communications intensive tasks as packet switching, media control and management.
According to particular example embodiments, the system 1000 uses memory 1003 to store data and program instructions and maintained a local side cache. The program instructions may control the operation of an operating system and/or one or more applications, for example. The memory or memories may also be configured to store received metadata and batch requested metadata.
Because such information and program instructions may be employed to implement the systems/methods described herein, the present invention relates to tangible, machine readable media that include program instructions, state information, etc. for performing various operations described herein. Examples of machine-readable media include hard disks, floppy disks, magnetic tape, optical media such as CD-ROM disks and DVDs; magneto-optical media such as optical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM) and programmable read-only memory devices (PROMs). Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.
According to various embodiments, a potentially infinite variety of filters or modifications may be applied to digital media content. Although various examples have been describe elsewhere in this application, some additional examples are provided here to provide additional context. When a person is detected with arms outstretched, visual elements such as angel wings, bat wings, butterfly wings, plane wings and engines, or a jetpack with exhaust fumes may be added. When a person is detected in a leaning posture, visual elements such as a dinosaur tail, a squirrel tail, or a raccoon tail may be added. When a person is detecting standing with hands on hips, visual elements may be added to replace the person's clothing with a superhero costume or to add a cape to the person's existing attire. When a person is detected as yelling, for instance with hands cupped around the mouth, visual elements may be added to depict a megaphone, flames, or a speech bubble near the person's mouth. Depending on a person's pose, visual elements may be added to replace a person's clothing or depict a person's body as a skeleton. When a person is detected as standing in a body builder's pose, the person's body may be replaced with one exhibiting more muscles or deformed to appear to exhibit more muscles. When a person is detected as having a hand over a mouth, visual elements may be added to make the person appear to be underwater as a scuba diver or mermaid. When a person is detect as leaning forward in a flying position, visual elements may be added to make the person appear to be a flying angel or super hero. For instance, a person's legs may be moved to make the person appear to be not supported by the ground. When a person is detected with arms uplifted, visual elements may be added to cause rainbows, money, or angels to appear over the person. When a person is detected with hands arranged in a boxing pose, visual elements may be added to make the person appear to be wearing boxing gloves or holding a weapon. A person's facial features or body may be modified to make the person appear to have the head or body of an animal, a fruit, a robot, or some other such object. A person's facial features may be detected and then used to select a corresponding emoticon, which then may be used to replace the person's head. When a person is detected as walking a dog, the dog's head and the person's head may be swapped. A person may be made to appear much thinner, heavier, more muscular, less muscular, or wavier than in reality. Motion blur may be added to make a person appear to be spinning very quickly. The preceding examples provide additional context about the types of visual modifications that could be made, but a potentially infinite variety of visual modifications may be provided according to various embodiments.
Although particular features have been described as part of each example in the present disclosure, any combination of these features or additions of other features are intended to be included within the scope of this disclosure. Accordingly, the embodiments described herein are to be considered as illustrative and not restrictive. Furthermore, although many of the components and processes are described above in the singular for convenience, it will be appreciated by one of skill in the art that multiple components and repeated processes can also be used to practice the techniques of the present disclosure.
While the present disclosure has been particularly shown and described with reference to specific embodiments thereof, it will be understood by those skilled in the art that changes in the form and details of the disclosed embodiments may be made without departing from the spirit or scope of the invention. Specifically, there are many alternative ways of implementing the processes, systems, and apparatuses described. It is therefore intended that the invention be interpreted to include all variations and equivalents that fall within the true spirit and scope of the present invention.

Claims

What is claimed is:

1. A method comprising:

analyzing a visual digital media item with a processor to identify one or more characteristics associated with the visual digital media item, wherein the characteristics include a physical object represented in the visual digital media item;

based on the identified characteristics, selecting a visual digital media modification from a plurality of visual digital media modifications for application to the visual digital media item; and

providing the selected visual digital media modification for presentation in a user interface for selection by a user.

2. The method recited in claim 1, wherein the visual digital media item is a video stream.

3. The method recited in claim 2, the raw video stream is a live camera view captured via a camera.

4. The method recited in claim 1, wherein the visual digital media item includes a surround view of the object, wherein the surround view of the object includes spatial information, scale information, or a plurality of different viewpoint images of the object.

5. The method recited in claim 1, wherein the characteristics include structure information indicating a physical context in which the physical object is positioned.

6. The method recited in claim 1, wherein the characteristics include pose information indicating an attitude or position associated with the physical object.

7. The method recited in claim 1, wherein the characteristics include movement information indicating a degree of velocity or acceleration of the object.

8. The method recited in claim 1, wherein the visual digital media modification includes a virtual object positioned within the visual digital media item.

9. The method recited in claim 1, wherein the visual digital media modification includes an artificial light source that appears to be blocked by the physical object in the visual digital media item.

10. The method recited in claim 1, wherein the visual digital media modification includes motion blur indicating movement associated with the object or a change to the color of a portion of the visual digital media item.

11. The method recited in claim 1, wherein the object is a human being, and wherein the visual digital media modification includes a text bubble appearing in proximity to a face.

12. The method recited in claim 1, the method further comprising:

receiving the visual digital media item at a server via a network from a client device, wherein providing the visual digital media modification for presentation in a user interface includes transmitting a message via the network to the client device.

13. The method recited in claim 1, wherein analyzing the visual digital media item comprises transmitting the visual digital media item to a server via a network from a client device and receiving a response message at the client device, the response message identifying the one or more characteristics.

14. The method recited in claim 1, wherein the visual digital media modification is automatically applied to the visual digital media item when it is determined that the visual digital media modification meets one or more designated criteria.

15. The method recited in claim 1, wherein the one or more designated criteria includes determining that the object is a person raising a hand.

16. A computing device comprising:

a camera configured to capture a visual digital media item;

a processor configured to identify one or more characteristics associated with the visual digital media item, wherein the characteristics include a physical object represented in the visual digital media item and to select, based on the identified characteristics, visual digital media modification from a plurality of visual digital media modifications for application to the visual digital media item; and

a display screen configured to present a user interface providing the selected visual digital media modification for user selection.

17. The computing device recited in claim 16, wherein the visual digital media item includes a surround view of the object, wherein the surround view of the object includes spatial information, scale information, and a plurality of different viewpoint images of the object.

18. The computing device recited in claim 16, wherein identifying the object comprises transmitting the visual digital media item to a server via a network from the computing device and receiving a response message at the computing device, the response message identifying the one or more characteristics.

19. The computing device recited in claim 16, wherein the characteristics include information selected from the group consisting of: structure information indicating a physical context in which the physical object is positioned, pose information indicating an attitude or position associated with the physical object, and movement information indicating a degree of velocity or acceleration of the object.

20. One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:

providing the visual digital media modification for presentation in a user interface for selection by a user.