US20130321586A1

US20130321586A1 - Cloud based free viewpoint video streaming

Info

Publication number: US20130321586A1
Application number: US13/588,917
Authority: US
Inventors: Adam Kirk; Patrick Sweeney; Don Gillett; Neil Fishman; Kanchan Mitra; Amit Mital; David Harnett; Yaron Eshet; Simon Winder; David Eraker
Original assignee: Microsoft Corp
Current assignee: Microsoft Technology Licensing LLC
Priority date: 2012-05-31
Filing date: 2012-08-17
Publication date: 2013-12-05
Also published as: US20130321590A1; US20130321566A1; US20130321418A1; US9251623B2; US8917270B2; US20130321589A1; US20130321410A1; US20130321575A1; US20130321396A1; US9846960B2; US20130321593A1; US9256980B2; US20130321413A1

Abstract

Cloud based FVV streaming technique embodiments presented herein generally employ a cloud based FVV pipeline to create, render and transmit FVV frames depicting a captured scene as would be viewed from a current synthetic viewpoint selected by an end user and received from a client computing device. The FVV frames use a similar level of bandwidth as a conventional streaming movie would consume. To change viewpoints, a new viewpoint is sent from the client to the cloud, and a new streaming movie is initiated from the new viewpoint. Frames associated with that viewpoint are created, rendered and transmitted to the client until a new viewpoint request is received.

Description

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of and priority to provisional U.S. patent application Ser. No. 61/653,983 filed May 31, 2012.

BACKGROUND

A given video generally includes one or more scenes, where each scene in the video can be either relatively static (e.g., the objects in the scene do not substantially change or move over time) or dynamic (e.g., the objects in the scene substantially change and/or move over time). In a traditional video the viewpoint of each scene is chosen by the director when the video is recorded/captured and this viewpoint cannot be controlled or changed by an end user while they are viewing the video. In other words, in a traditional video the viewpoint of each scene is fixed and cannot be modified when the video is being rendered and displayed. In a free viewpoint video (FVV) an end user can interactively control and change their viewpoint of each scene at will while they are viewing the video. In other words, in a FVV each end user can interactively generate synthetic (i.e., virtual) viewpoints of each scene on-the-fly while the video is being rendered and displayed. This creates a feeling of immersion for any end user who is viewing a rendering of the captured scene, thus enhancing their viewing experience.

SUMMARY

Cloud based free viewpoint video (FVV) streaming technique embodiments described herein generally involve generating a FVV that provides a consistent and manageable amount of data to a client despite the large amounts of data typically demanded to create and render the FVV. In one general embodiment, this is accomplished by first capturing a scene using an arrangement of sensors. This sensor arrangement includes a plurality of sensors that generate a plurality of streams of sensor data, where each stream represents the scene from a different geometric perspective. These streams of sensor data are input and calibrated, and then scene proxies are generated from the calibrated streams of sensor data. The scene proxies geometrically describe the scene as a function of time. Next, a current synthetic viewpoint of the scene is received from a client computing device via a data communication network. This current synthetic viewpoint was selected by an end user of the client computing device. Once a current synthetic viewpoint is received, a sequence of frames is generated using the scene proxies. Each frame of the sequence depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene, and is transmitted to the client computing device via the data communication network for display to the end user of the client computing device.
From the perspective of a client computing device, a FVV produced as described above is played in one general embodiment as follows. A request is received from an end user to display a FVV selection user interface screen that allows the end user to select a FVV available for playing. This FVV selection user interface screen is displayed on a display device, and an end user FVV selection is input. The end user FVV selection is then transmitted to a server via a data communication network. The client computing device then receives an instruction from the server via the data communication network to instantiate end user controls appropriate for the type of FVV selected. In response, an appropriate FVV control user interface is provided to the end user. The client computing device then monitors end user inputs via the FVV control user interface, and whenever an end user viewpoint navigation input is received, it is transmitted to the server via the data communication network. FVV frames are then received from the server. Each FVV frame depicts at least a portion of the captured scene as it would be viewed from the last viewpoint the end user input, and is displayed on the aforementioned display device as it is received.
It is noted that this Summary is provided to introduce a selection of concepts, in a simplified form, that are further described hereafter in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the free viewpoint video processing pipeline technique embodiments described herein will become better understood with regard to the following description, appended claims, and accompanying drawings where:

FIG. 1 is a diagram illustrating an exemplary embodiment, in simplified form, of the various stages in a free viewpoint video (FVV) pipeline.

FIG. 2 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for generating a FVV of a scene.

FIG. 3 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for capturing and inputting scene data.

FIG. 4 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for calibrating the streams of sensor data which are generated by an arrangement of sensors that is being used to capture the scene.

FIG. 5 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for processing the calibrated streams of sensor data to generate scene proxies which geometrically describes the scene as a function of time.

FIG. 6 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for using a point cloud 3D reconstruction method to generate 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of depth map images of the scene.

FIG. 7 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of infrared images of the scene.

FIG. 8 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of color images of the scene.

FIG. 9 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for processing the scene proxies to render a FVV.

FIG. 10 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for implementing the rendering of FIG. 10.

FIG. 11 is a diagram illustrating the various degrees of viewpoint navigation freedom that can be supported by the FVV pipeline.

FIG. 12 is a diagram illustrating an exemplary embodiment, in simplified form, of a continuum of the various image-based rendering methods which can be employed by the FVV pipeline.

FIGS. 13A-B are a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for presenting a FVV to an end user.

FIG. 14 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for presenting a FVV to an end user, where the end user can temporally control the playback of the FVV.

FIG. 15 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for presenting a FVV to an end user, where the end user can temporally control the playback of the FVV to play it in reverse.

FIG. 16 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for presenting a FVV to an end user, where the end user can temporally control the playback to pause and restart the FVV.

FIGS. 17A-B are a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for rendering a FVV that involves predicting the next new viewpoint to be requested by the end user.

FIG. 18 is a diagram illustrating a simplified example of a general-purpose computer system on which various embodiments and elements of the cloud based FVV streaming technique, as described herein, may be implemented.

DETAILED DESCRIPTION

In the following description of cloud based free viewpoint video (FVV) streaming technique embodiments (hereafter sometimes simply referred to as streaming technique embodiments) reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific embodiments in which the streaming technique can be practiced. It is understood that other embodiments can be utilized and structural changes can be made without departing from the scope of the streaming technique.
It is also noted that for the sake of clarity specific terminology will be resorted to in describing the streaming technique embodiments described herein and it is not intended for these embodiments to be limited to the specific terms so chosen. Furthermore, it is to be understood that each specific term includes all its technical equivalents that operate in a broadly similar manner to achieve a similar purpose. Reference herein to “one embodiment”, or “another embodiment”, or an “exemplary embodiment”, or an “alternate embodiment”, or “one implementation”, or “another implementation”, or an “exemplary implementation”, or an “alternate implementation” means that a particular feature, a particular structure, or particular characteristics described in connection with the embodiment or implementation can be included in at least one embodiment of the streaming technique. The appearances of the phrases “in one embodiment”, “in another embodiment”, “in an exemplary embodiment”, “in an alternate embodiment”, “in one implementation”, “in another implementation”, “in an exemplary implementation”, “in an alternate implementation” in various places in the specification are not necessarily all referring to the same embodiment or implementation, nor are separate or alternative embodiments/implementations mutually exclusive of other embodiments/implementations. Yet furthermore, the order of process flow representing one or more embodiments or implementations of the streaming technique does not inherently indicate any particular order nor imply any limitations of the streaming technique.
The term “sensor” is used herein to refer to any one of a variety of scene-sensing devices which can be used to generate a stream of sensor data that represents a given scene. Generally speaking and as will be described in more detail hereafter, the streaming technique embodiments described herein employ a plurality of sensors which can be configured in various arrangements to capture a scene, thus allowing a plurality of streams of sensor data to be generated each of which represents the scene from a different geometric perspective. Each of the sensors can be any type of video capture device (e.g., any type of video camera), or any type of audio capture device, or any combination thereof. Each of the sensors can also be either static (i.e., the sensor has a fixed spatial location and a fixed rotational orientation which do not change over time), or moving (i.e., the spatial location and/or rotational orientation of the sensor change over time). The streaming technique embodiments described herein can employ a combination of different types of sensors to capture a given scene.
The term “baseline” is used herein to refer to a ratio of the actual physical distance between a given pair of VCDs to the average of the actual physical distance from each sensor in the pair to the viewpoint of the scene. When this ratio is larger than a prescribed value the pair of sensors is referred to herein as a “wide baseline stereo pair of sensors”. When this ratio is smaller than the prescribed value the pair of sensors is referred to herein as a “narrow baseline stereo pair of sensors”.
The term “server” is used herein to refer to one or more server computing devices operating in a cloud infrastructure so as to provide FVV services to a client computer over a data communication network.

1.0 Cloud Based Free Viewpoint Video (FVV) Streaming

The creation and playback of FVV involves working with a substantial amount of data. Firstly, a scene is simultaneously recorded from many different perspectives using sensors such as RGB cameras. Second, this data is processed to extract three dimensional (3D) geometric information in the form of scene proxies using, for example, 3D Reconstruction (3DR) algorithms. Finally, the original data and geometric proxies is recombined during rendering, using Image Based rendering (IBR) algorithms to generate synthetic viewpoints.
Moreover, the amount of data may vary considerably from one FVV to another FVV due to difference in the number of sensors used to record the scene, the length of the FVV, the type of 3DR algorithms used to process the data, and the type of IBR algorithm used to generate synthetic views of the scene. In addition, there exists a wide variety of different combinations of both bandwidth and local processing power that can be used for viewing at a client.
One way in which a FVV can be transferred from a server to a client over a data communication network (such as the Internet or a proprietary intranet) is to combine the 3D geometry and other data for a specific viewpoint to produce a single image or video frame on the server, and then to transmit this frame from the server to the client. The frame is then displayed by the client in a normal manner. This pre-computed frame transmission approach has the advantage of providing a consistent and manageable amount of data to a client despite the large amounts of data demanded to create and render a FVV, and the fact that the amount of data can be constantly changing. In other words, the FVV data stays on a server (or servers) in the cloud, and even clients with limited processing power and/or limited available bandwidth can receive and display a FVV. More particularly, an advantage of creating a cloud based streaming FVV that represents what would be seen from a specific viewpoint is that FVV's can be commercially deployed to end users and use a similar level of bandwidth as a conventional streaming movie would consume. This approach will be referred to herein as cloud based FVV streaming.
To change viewpoints, a new (typically user specified) viewpoint is sent from the client to the server, and a new stream of video data is initiated from the new viewpoint. Frames associated with that viewpoint are created, rendered and transmitted to the client until a new viewpoint request is received.
Cloud based FVV streaming technique embodiments described herein generally employ a cloud based FVV pipeline to create, render and transmit FVV frames depicting a captured scene as would be viewed from a current synthetic viewpoint received from a client. An exemplary FVV pipeline will be now be described. It is noted, however, that cloud based FVV streaming technique embodiments described herein are not limited to only the exemplary FVV pipeline to be described. Rather, other FVV pipelines can also be employed to create and render video frames in response to a viewpoint request, as desired.

1.1 Exemplary FVV Pipeline for Cloud Based FVV Streaming

Generally speaking, the exemplary FVV pipeline described here involves generating an FVV of a given scene and presenting the FVV to one or more end users. Generally speaking and as will be appreciated from the more detailed description that follows, the exemplary FVV pipeline enables optimal viewpoint navigation for up to six degrees of viewpoint navigation freedom. Furthermore, this exemplary FVV pipeline does not rely upon having to constrain the pipeline in order to produce a desired visual result. In other words, the pipeline eliminates the need to place constraints in order to generate various synthetic viewpoints of the scene which are photo-realistic and thus are free of discernible artifacts. More particularly and by way of example but not limitation, the pipeline eliminates having to constrain the arrangement of the sensors that are used to capture the scene. Accordingly, the pipeline is operational with any arrangement of sensors. For example, the pipeline eliminates having to constrain the number or types of sensors that are used to capture the scene. Accordingly, the pipeline is operational with any number of sensors and all types of sensors. The pipeline also eliminates having to constrain the number of degrees of viewpoint navigation freedom that are provided during the rendering and end user viewing of the captured scene. Accordingly, the pipeline can produce visual results having as many as six degrees of viewpoint navigation freedom. Further, the pipeline eliminates having to constrain the complexity or composition of the scene that is being captured (e.g., neither the environment(s) in the scene, nor the types of objects in the scene, nor the number of people of in the scene, among other things has to be constrained). Accordingly, the pipeline is operational with any type of scene, including both relatively static and dynamic scenes.
Yet further, the pipeline does not rely upon having to use a specific 3D reconstruction method to generate a 3D reconstruction of the captured scene. Accordingly, the pipeline supports the use of any one or more 3D reconstruction methods and therefore provides the freedom to use whatever 3D reconstruction method(s) produces the desired visual result (e.g., the highest degree of photo-realism for the particular scene being captured and the desired number of degrees of viewpoint navigation freedom) based on the particular characteristics of the streams of sensor data that are generated by the sensors (e.g., based on factors such as the particular number and types of sensors that are used to capture the scene, and the particular arrangement of these sensors that is used), along with other current pipeline conditions.
The exemplary pipeline also does not rely upon having to use a specific image-based rendering method during the rendering of a frame of the captured scene. Accordingly, the pipeline supports the use of any image-based rendering method and therefore provides the freedom to use whatever image-based rendering method(s) produces the desired visual result based on the particular characteristics of the streams of sensor data that are generated by the sensors, along with other current pipeline conditions. By way of example but not limitation, in an exemplary situation where just two video capture devices are used to capture a scene, an image-based rendering method that renders a lower fidelity 3D geometric proxy of the captured scene may produce an optimally photo-realistic visual result when the end user's viewpoint is close to the axis of one of the video capture devices (such as with billboards). In another exemplary situation where 36 video capture devices configured in a circular arrangement are used to capture a scene, a conventional image warping/morphing image-based rendering method may produce an optimally photo-realistic visual result. In yet another exemplary situation where 96 video capture devices configured in either a 2D (two-dimensional) or 3D array arrangement are used to capture a scene, a conventional view interpolation image-based rendering method may produce an optimally photo-realistic visual result. In yet another exemplary situation where an even larger number of video capture devices is used, a conventional lumigraph or light field image-based rendering method may produce an optimally photo-realistic visual result.
It will thus be appreciated that the exemplary pipeline results in a flexible, robust and commercially viable next generation FVV processing pipeline that meets the needs of today's various creative video producers and editors. By way of example but not limitation and as will be appreciated from the more detailed description that follows, the pipeline is applicable to various types of video-based media applications such as consumer entertainment (e.g., movies, television shows, and the like) and videoconference/telepresence, among others. The pipeline can support a broad range of features that provide for the capture, processing, storage, rendering and distribution of any type of FVV that can be generated. Various implementations of the pipeline are possible, where each different implementation supports a different type of FVV. Exemplary types of supported FVV are described in more detail hereafter.
Additionally, the pipeline allows any one or more parameters to be freely modified without introducing artifacts into the FVV. This allows the photo-realism of the FVV that is presented to each end user to be maximized (i.e., the artifacts are minimized) regardless of the characteristics of the various sensors that are used to capture the scene, and the characteristics of the various streams of sensor data that are generated by the sensors. Exemplary pipeline parameters which can be modified include, but are not limited to, the following. The number and types of sensors that are used to capture the scene can be modified. The arrangement of the sensors can also be modified. Which if any of the sensors is static and which is moving can also be modified. The complexity and composition of the scene can also be modified. Whether the scene is relatively static or dynamic can also be modified. The 3D reconstruction methods and image-based rendering methods that are used can also be modified. The number of degrees of viewpoint navigation freedom that are provided during the rendering and end user viewing of the captured scene can also be modified.

1.1.1 Pipeline Stages Overview and Exemplary Process

FIG. 1 illustrates an exemplary embodiment, in simplified form, of the various stages in a FVV processing pipeline. As exemplified in FIG. 1, the FVV processing pipeline 100 starts with a capture stage 102 during which, and generally speaking, the following actions take place. An arrangement of sensors is used to capture a given scene, where the arrangement includes a plurality of sensors and generates a plurality of streams of sensor data each of which represents the scene from a different geometric perspective. These streams of sensor data are input from the sensors and calibrated in a manner which will be described in more detail hereafter. The calibrated streams of sensor data are then output to a processing stage 104. As described heretofore, the exemplary pipeline supports the use of various types, various numbers and various combinations of sensors which can be configured in various arrangements, including both 2D and 3D arrangements, where each of the sensors can be either static or moving.
Referring again to FIG. 1, the processing stage 104 of the FVV processing pipeline 100 inputs the calibrated streams of sensor data and processes them to generate scene proxies which geometrically describes the captured scene as a function of time. The scene proxies are then output to a storage stage 106 where they are stored in a memory via conventional means. As will be described in more detail hereafter, the scene proxies are generated using one or more different 3D reconstruction methods which extract 3D geometric information from the calibrated streams of sensor data. The particular 3D reconstruction methods that are used and the particular manner in which each scene proxy is generated are determined based on a periodic analysis of a set of current conditions.
Referring again to FIG. 1, a rendering stage 108 of the FVV processing pipeline 100 inputs scene proxies from the storage stage 106 and processes them to generate frames of the captured scene as would be viewed from a current synthetic viewpoint. The rendering is designed to maximize the photo-realism of the frame produced. Generally speaking, the aforementioned frames are generated using one or more different image-based rendering methods. The particular image-based rendering methods that are used and the particular manner in which each frame is generated are determined based on a periodic analysis of the set of current conditions. The frames of the captured scene are then output to a user viewing experience stage 110.
Referring again to FIG. 1, the user viewing experience stage 110 of the FVV processing pipeline 100 generally provides one or more end users with the ability to view the frames of the captured scene, as the scene would be viewed from a current synthetic viewpoint, on a display device and to navigate/control this viewpoint on-the-fly at will. In other words, the user viewing experience stage 110 provides each end user with the ability to continuously and interactively navigate/control their viewpoint of the scene that is being displayed on their display device. The user viewing experience stage 110 may also provide each end user with the ability to interact temporally with the FVV at will, as will be explained in more detail later.
More particularly, each end user can interactively navigate their viewpoint of the scene via a client computing device 112 associated with that user. Each time an end user chooses a different viewpoint, this new viewpoint is provided to the user viewing experience stage 110 by the user's client computing device 112 via a data communication network 114 that the end user's client computing devices is connected to (such as the Internet or a proprietary intranet). This transfer of the new viewpoint is done in a conventional manner consistent with the network being employed. The user viewing experience stage 110 receives and forwards the new viewpoint to the rendering stage 108, which will modify the current synthetic viewpoint of the scene accordingly and produce frames of the captured scene as would be viewed from the new synthetic viewpoint. These frames are then provided in turn to the user viewing experience stage 110. The user viewing experience stage 110 receives each frame and provides it the user's client computing device 112 via the aforementioned network 114. This transfer of the frames is also done in a conventional manner consistent with the network being employed.
In situations where FVVs have been stored for future playback, each end user can also interact temporally to control the playback of the FVV, and based on this temporal control the rendering stage 108 will provide FVV frames starting with the frame that corresponds to the last user-specified temporal location in the FVV.
The foregoing FVV processing pipeline can be employed by the cloud based FVV streaming technique embodiments described herein to generate a free viewpoint video (FVV) of a scene. More particularly, in one general implementation outlined in FIG. 2, a computer (such as any of the computing devices described in the Exemplary Operating Environments to follow) is used to perform the following actions. First, the aforementioned scene is captured using an arrangement of sensors (block 200). This sensor arrangement includes a plurality of sensors that generate a plurality of streams of sensor data, where each stream represents the scene from a different geometric perspective. These streams of sensor data are input and calibrated (block 202), and then scene proxies are generated from the calibrated streams of sensor data (block 204). The scene proxies geometrically describe the scene as a function of time. Next, a current synthetic viewpoint of the scene is received from a client computing device via a data communication network (block 206). It is noted that this current synthetic viewpoint was selected by an end user of the client computing device. Once a current synthetic viewpoint is received, a sequence of frames is generated using the scene proxies (block 208). Each frame of the sequence depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene, and in block 210 is transmitted to the client computing device via the data communication network for display to the end user of the client computing device.
It is noted, however, that in situations where a current synthetic viewpoint is not selected by the end user prior to playing a FVV, the performance of blocks 208 and 210 are deferred until the viewpoint is selected, and in the meantime, a sequence of frames is generated using the scene proxies, where each frame depicts at least a portion of the scene as viewed from a prescribed default viewpoint of the scene. These frames are transmitted in turn to the client computing device via the data communication network for display to the end user of the client computing device.
It is further noted that, as indicated previously, in one embodiment the scene proxies are stored as they are generated. This feature can allow complete FVVs to be recorded for playback at a future time.

1.1.2 Supported FVV Types

As noted heretofore, various implementations of the pipeline are possible, where each different implementation supports a different type of FVV and a different user viewing experience. As will now be described in more detail, each of these different implementations differs in terms of the user viewing experience it provides, its latency characteristics (i.e., how rapidly the streams of sensor data have to be processed through the FVV processing pipeline), its storage characteristics, and the types of computing device hardware it necessitates.
Referring again to FIG. 1, one implementation of the pipeline supports non-live FVVs, which corresponds to a situation where the streams of sensor data that are generated by the sensors are captured and processed for future playback. As such, there is effectively an unlimited amount of time available for the processing stage 104. This allows an FVV producer to optionally manually “touch-up” the streams of sensor data that are input during the capture stage 102, and also optionally manually remove any 3D reconstruction artifacts that are introduced in the processing stage 104. This particular implementation is referred to hereafter as the “recorded FVV implementation”. Exemplary types of video-based media that work well in the recorded FVV implementation include movies, documentaries, sitcoms and other types of television shows, music videos, digital memories, and the like. Another exemplary type of video-based media that works well in the recorded FVV implementation is the use of special effects technology where synthetic objects are realistically modeled, lit, shaded and added to a pre-captured scene.
Referring again to FIG. 1, another implementation of the pipeline supports unidirectional (i.e., one-way) live FVV, which corresponds to a situation where the streams of sensor data that are being generated by the sensors are concurrently captured 102 and processed 104; and the resulting 3D scene proxy of the captured scene is stored and rendered into frames, and then transmitted in a one-to-many manner on-the-fly (i.e., live) to one or more end users. As such, each end user can view the scene live (i.e., each user can view the scene at substantially the same time it is being captured 102). This particular implementation is referred to hereafter as the “unidirectional live FVV implementation”. Exemplary types of video-based media that work well in the unidirectional live FVV implementation include sporting events, news programs, live concerts, and the like.
Referring again to FIG. 1, yet another implementation of the supports bidirectional (i.e., two-way) live FVV such as that which is associated with various videoconferencing/telepresence applications. This particular implementation is referred to hereafter as the “bidirectional live FVV implementation”. This bidirectional live FVV implementation is generally the same as the unidirectional live FVV implementation with the following exception. In the bidirectional live FVV implementation a computing device at each physical location that is participating in a given videoconferencing/telepresence session is able to concurrently capture streams of sensor data that are being generated by sensors which are capturing a local scene and processing these locally captured streams of sensor data into scene proxies, storing the proxies, render the proxies into frames, and transmitting the resulting FVV frames of the local scene in a one-to-many manner on the fly to the other physical locations that are participating in the session.
Referring again to FIG. 1, it will be appreciated that in the unidirectional and bidirectional live FVV implementations of the pipeline, in order for an end user to be able to view the scene live the capture, processing, storing, rendering and frame distribution have to be completed within a prescribed very short period of time. In an exemplary embodiment of the pipeline this period of time is one thirtieth of a second per frame.

1.1.3 FVV Capture and Processing Stages

This section provides a more detailed description of the capture and processing stages of the FVV processing pipeline. The exemplary pipeline generally employs a plurality of sensors which are configured in a prescribed arrangement to capture a given scene. The pipeline is operable with any type of sensor, any number (two or greater) of sensors, any arrangement of sensors (where this arrangement can include a plurality of different geometries and different geometric relationships between the sensors), and any combination of different types of sensors. The pipeline is also operable with both static and moving sensors. A given sensor can be any type of video capture device (examples of which are described in more detail hereafter), or any type of audio capture device (such as a microphone, or the like), or any combination thereof. Each video capture device generates a stream of video data which includes a stream of images (also known as and referred to herein as “frames”) of the scene from the specific geometric perspective of the video capture device. Similarly, each audio capture device generates a stream of audio data representing the audio emanating from the scene from the specific geometric perspective of the audio capture device.
Exemplary types of video capture devices that can be employed include, but are not limited to, the following. A given video capture device can be a conventional visible light video camera which generates a stream of video data that includes a stream of color images of the scene. A given video capture device can also be a conventional light-field camera (also known as a “plenoptic camera”) which generates a stream of video data that includes a stream of color light field images of the scene. A given video capture device can also be a conventional infrared structured-light projector combined with a conventional infrared video camera that is matched to the projector, where this projector/camera combination generates a stream of video data that includes a stream of infrared images of the scene. This projector/camera combination is also known as a “structured-light 3D scanner”. A given video capture device can also be a conventional monochromatic camera which generates a stream of video data that includes a stream of monochrome images of the scene. A given video capture device can also be a conventional time-of-flight camera which generates a stream of video data that includes both a stream of depth map images of the scene and a stream of color images of the scene. For simplicity sake, the term “color camera” is sometimes used herein to refer to any type of video capture device that generates color images of the scene.
It will be appreciated that variability in factors such as the composition and complexity of a given scene, and each end user's viewpoint navigation, among other factors, can impact the determination of how many sensors to use to capture the scene, the particular type(s) of sensors to use, and the particular arrangement of the sensors. The exemplary pipeline generally employs a minimum of one sensor which generates color image data for the scene, along with one or more other sensors that can be used in combination to generate 3D geometry data for the scene. In situations where an outdoor scene is being captured or the sensors are located far from the scene, it is advantageous to capture the scene using both a wide baseline stereo pair of color cameras and a narrow baseline stereo pair of color cameras. In situations where an indoor scene is being captured, it is advantageous to capture the scene using a narrow baseline stereo pair of sensors both of which generate video data that includes a stream of infrared images of the scene in order to eliminate the dependency on scene lighting variables.
Generally speaking, it is advantageous to increase the number of sensors being used as the complexity of the scene increases. In other words, as the scene becomes more complex (e.g., as additional people are added to the scene), the use of additional sensors serves to reduce the number of occluded areas within the scene. It may also be advantageous to capture the entire scene using a given arrangement of static sensors, and at the same time also capture a specific higher complexity region of the scene using one or more additional moving sensors. In a situation where a large number of sensors is used to capture a complex scene, different combinations of the sensors can be used during the processing stage of the FVV processing pipeline (e.g., a situation where a specific sensor is part of both a narrow baseline stereo pair and a different wide baseline stereo pair involving a third sensor).
FIG. 3 illustrates an exemplary embodiment, in simplified form, of a process for capturing and inputting scene data. As exemplified in FIG. 3, the process starts in block 300 with using an arrangement of sensors to capture the scene, where the arrangement includes a plurality of sensors and generates a plurality of streams of sensor data each of which represents the scene from a different geometric perspective. The streams of sensor data are then input (block 302). It will be appreciated that a given stream of sensor data will include video data whenever the sensor that generated the stream is a video capture device. A given stream of sensor data will include audio data whenever the sensor that generated the stream is an audio capture device. A given stream of sensor data will include both video and audio data whenever the sensor that generated the stream is a combined video and audio capture device.
FIG. 4 illustrates an exemplary embodiment, in simplified form, of a process for calibrating the streams of sensor data which are generated by the arrangement of sensors. As exemplified in FIG. 4, the process starts in block 400 with determining the number of sensors in the arrangement of sensors that is being used to capture the scene. Intrinsic characteristics of each of the sensors are then determined (block 402). Exemplary intrinsic characteristics which can be determined for a given sensor include the sensor type, the sensor's frame rate, the sensor's shutter speed, the sensor's mosaic pattern, the sensor's white balance, the bit depth and pixel resolution of the images that are generated by the sensor, the focal length of the VCD's lens, the principal point of the VCD's lens, the VCD's skew coefficient, the distortions of the VCD's lens, the VCD's field of view, among others. It will be appreciated that knowing such intrinsic characteristics for each of the sensors allows the FVV processing pipeline to understand the governing physics and optics of each of the sensors. Extrinsic characteristics of each of the sensors at each point in time during the capture of the scene are also determined (block 404). Exemplary extrinsic characteristics which can be determined for a given sensor include the sensor's current rotational orientation (i.e., the direction that the sensor is currently pointing), the sensor's current spatial location (i.e., the sensor's current location within the arrangement), whether the sensor is static or moving, the current geometric relationship between the sensor and each of the other sensors in the arrangement (i.e., the sensor's current position relative to each of the other sensors), the position of the sensor relative to the scene, and whether or not the sensor is genlocked (i.e., temporally synchronized) with the other sensors in the arrangement, among others. The determination of the intrinsic and extrinsic characteristics of each of the sensors can be made using various conventional methods, examples of which will be described in more detail hereafter. The knowledge of the number of sensors in the arrangement, and the intrinsic and extrinsic characteristics of each of the sensors, is then used to temporally and spatially calibrate the streams of sensor data (block 406).
As is appreciated in the art of video recording, the intrinsic and extrinsic characteristics of each of the sensors in the arrangement are commonly determined by performing one or more calibration procedures which calibrate the sensors, where these procedures are specific to the particular types of sensors that are being used to capture the scene, and the particular number and arrangement of the sensors. In the unidirectional and bidirectional live FVV implementations of the pipeline, the calibration procedures are performed and the streams of sensor data which are generated thereby are input before the scene capture. In the recorded FVV implementation of the pipeline, the calibration procedures can be performed and the streams of sensor data which are generated thereby can be input either before or after the scene capture. Exemplary calibration procedures will now be described.
In a situation where the sensors that are being used to capture the scene are genlocked and include a combination of color cameras, sensors which generate a stream of infrared images of the scene, and one or more time-of-flight cameras, and this combination of cameras is arranged in a static array, the cameras in the array can be calibrated and the intrinsic and extrinsic characteristics of each of the cameras can be determined in the following manner. A stream of calibration data can be input from each of the cameras in the array while a common physical feature (such as a ball, or the like) is internally illuminated with an incandescent light (which is visible to all of the cameras) and moved throughout the scene. These streams of calibration data can then be analyzed using conventional methods to determine both an intrinsic and extrinsic calibration matrix for each of the cameras.
In another situation where the sensors that are being used to capture the scene include a plurality of color cameras which are arranged in a static array, the cameras in the array can be calibrated and the intrinsic and extrinsic characteristics of each of the cameras can be determined in the following manner. A stream of calibration data can be input from each camera in the array while it is moved around the scene but in close proximity to its static location (thus allowing each camera in the array to view overlapping parts of the static background of the scene). After the scene is captured by the static array of color cameras and the streams of sensor data generated thereby are input, the streams of sensor data can be analyzed using conventional methods to identify features in the scene, and these features can then be used to calibrate the cameras in the array and determine the intrinsic and extrinsic characteristics of each of the cameras by employing a conventional method (e.g., extrinsic characteristics can be determined using a structure-from-motion method).
In yet another situation where one or more of the sensors that are being used to capture the scene are moving sensors (such as when the spatial location of a given sensor changes over time, or when controls on a given sensor are used to optically zoom in on the scene while it is being captured (which is commonly done during the recording of sporting events, among other things)), each of these moving sensors can be calibrated and its intrinsic and extrinsic characteristics can be determined at each point in time during the scene capture by using a conventional background model to register and calibrate relevant individual images that were generated by the sensor. In yet another situation where the sensors that are being used to capture the scene include a combination of static and moving sensors, the sensors can be calibrated and the intrinsic and extrinsic characteristics of each of the sensors can be determined by employing conventional multistep calibration procedures.
In yet another situation where there is no temporal synchronization between the sensors that are being used to capture the scene and the arrangement of the sensors can randomly change over time (such as when a plurality of mobile devices are held up by different users and the sensors on these devices are used to capture the scene), the exemplary pipeline will both spatially and temporally calibrate the streams of sensor data generated by the sensors at all points in time during the scene capture before the streams are processed in the processing stage. In an exemplary embodiment of the pipeline technique this spatial and temporal calibration can be performed as follows. After the scene is captured and the streams of sensor data representing the scene are input, the streams of sensor data can be analyzed using conventional methods to separate the static and moving elements of the scene. The static elements of the scene can then be used to generate a background model. Additionally, the moving elements of the scene can be used to generate a global timeline that encompasses all of the sensors, and each image in each stream of sensor data is assigned a relative time. The intrinsic characteristics of each of the sensors can be determined by using conventional methods to analyze each of the streams of sensor data.
In an implementation of the exemplary pipeline where the capture stage of the FVV processing pipeline is directly connected to the sensors that are being used to capture the scene, the intrinsic characteristics of each of the sensors can also be determined by reading appropriate hardware parameters directly from each of the sensors. In another embodiment of the pipeline technique where the capture stage is not directly connected to the sensors but rather the streams of sensor data are pre-recorded and then imported into the capture stage, the number of sensors and various intrinsic properties of each of the sensors can be determined by analyzing the streams of sensor data using conventional methods.
FIG. 5 illustrates an exemplary embodiment, in simplified form, of a process for processing the calibrated streams of sensor data to generate the scene proxies. As exemplified in FIG. 5, the process starts in block 500 with monitoring and periodically analyzing a set of current pipeline conditions. The set of current pipeline conditions can include one or more conditions in the capture stage of the FVV processing pipeline such as the particular number of sensors that is being (or was) used to capture the scene, or the particular arrangement of these sensors that is being (or was) used, or one or more particular intrinsic characteristics of each of the sensors (e.g., the sensor type, among others), or one or more particular extrinsic characteristics of each of the sensors (e.g., the positioning of the sensor relative to the scene, and whether the sensor is static or moving, among others), or the like. The set of current pipeline conditions can also include one or more conditions in the processing stage of the pipeline such as whether the scene proxies are being generated on-the-fly, or being generated and stored for future playback (i.e., the particular type of FVV that is being processed in the pipeline and the speed at which the streams of sensor data have to be processed through the pipeline), or the like.
The set of current pipeline conditions can also include one or more conditions in the storage stage of the FVV processing pipeline such as the amount of storage space that is currently available to store the scene proxy. The set of current pipeline conditions can also include one or more conditions in the rendering stage of the pipeline such as the current viewpoint navigation information and temporal navigation information. In addition to including this viewpoint and temporal navigation information, the second set of current conditions is also generally associated with the specific implementation of the pipeline technique embodiments that is being used. The set of current pipeline conditions can also further include one or more conditions in the user viewing experience stage of the pipeline such as the particular type of display device the rendered frames are being displayed on, and the particular characteristics of the display device (e.g., its aspect ratio, its pixel resolution, and its form factor, among others).
Referring again to FIG. 5, after the analysis of the set of current pipeline conditions has been completed (block 500), the results of this analysis are then used to select one or more different 3D reconstruction methods which are matched to the current pipeline conditions (block 502). The selected 3D reconstruction methods are then used to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data (block 504). The 3D reconstructions of the scene and the results of the period analysis are then used to generate the scene proxy (block 506). The actions of blocks 500, 502, 504 and 506 are repeated for the duration of the scene (block 508, No). As such, it will be appreciated that the 3D reconstruction methods which are used, the types of 3D reconstructions of the scene which are generated, and thus the types of geometric proxy data in a scene proxy can change over time based upon changes in the current pipeline conditions. It will also be appreciated that the current pipeline conditions can be analyzed using different periodicities. By way of example but not limitation, in one embodiment of the exemplary pipeline the current pipeline conditions can be analyzed on a frame-by-frame basis (i.e., for each image in the streams of sensor data). In another embodiment of the pipeline, the current pipeline conditions can be analyzed using a periodicity of a prescribed number of sequential frames, where this number is greater than one. In yet another embodiment of the pipeline, the current pipeline conditions can be analyzed using a periodicity of a prescribed period of time.
It will thus be appreciated that the exemplary pipeline can use a wide variety of 3D reconstruction methods in various combinations, where the particular types of 3D reconstruction methods that are being used depend upon various current conditions in the FVV processing pipeline. Accordingly and as will be described in more detail hereafter, the scene proxies represent one or more types of geometric proxy data examples of which include, but are not limited to, the following. A scene proxy can include a stream of depth map images of the scene. A scene proxy can also include a stream of calibrated point cloud reconstructions of the scene. As is appreciated in the art of 3D reconstruction, these point cloud reconstructions are a low order geometric representation of the scene. A scene proxy can also include one or more types of high order geometric models such as planes, billboards, and existing (i.e., previously created) generic object models (e.g., human body models) which can be either modified, or animated, or both. A scene proxy can also include other high fidelity proxies such as a stream of mesh models of the scene, and the like. It will further be appreciated that since the particular 3D reconstruction methods that are used and the related manner in which a scene proxy is generated are based upon a period analysis (i.e., monitoring) of the various current conditions in the FVV processing pipeline, the 3D reconstruction methods that are used and the resulting types of data in the scene proxy can change over time based on changes in the pipeline conditions.
Generally speaking, for the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments described herein, due to the fact that the capture, processing, storage, rendering, user viewing experience stages of the FVV processing pipeline have to be completed within a prescribed very short period of time, the types of 3D reconstruction methods that can be used in these implementations are limited to high speed 3D reconstruction methods. By way of example but not limitation, in the unidirectional and bidirectional live FVV implementations of the pipeline, a scene proxy that is generated will include a stream of calibrated point cloud reconstructions of the scene, and may also include one or more types of higher order geometric models which can be either modified, or animated, or both. It will be appreciated that 3D reconstruction methods which can be implemented in hardware are also favored in the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments. The use of sensors which generate infrared images of the scene is also favored in the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments.
For the recorded FVV implementation of the pipeline, lower speed 3D reconstruction methods can be used. By way of example but not limitation, in the recorded FVV implementation of the pipeline, a scene proxy that is generated can include both a stream of calibrated point cloud reconstructions of the scene, as well as one or more higher fidelity geometric proxies of the scene (such as when the point cloud reconstructions are used to generate a stream of mesh models of the scene, among other possibilities). The recorded FVV implementation of the pipeline also allows a plurality of 3D reconstruction steps to be used in sequence when generating the scene proxy. By way of example but not limitation, consider a situation where a stream of calibrated point cloud reconstructions of the scene has been generated, but there are some noisy or error prone stereo matches present in these reconstructions that extend beyond a human silhouette boundary in the scene. It will be appreciated that these noisy or error prone stereo matches can lead to the wrong texture data appearing in the mesh models of the scene, thus resulting in artifacts in the rendered scene. These artifacts can be eliminated by running a segmentation process to separate the foreground from the background, and then points outside of the human silhouette can be rejected as outliers.
FIG. 6 illustrates an exemplary embodiment, in simplified form, of a process for using a point cloud 3D reconstruction method to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of depth map images of the scene (hereafter simply referred to as different depth map image streams). As exemplified in FIG. 6, whenever the calibrated streams of sensor data include a plurality of different depth map image streams (block 600, Yes), these different depth map image streams are merged into a stream of calibrated point cloud reconstructions of the scene (block 602). It is noted that these point cloud reconstructions are unordered and as such, overlaps may exist therein. Depending on the current pipeline conditions, the stream of calibrated point cloud reconstructions of the scene can then optionally be used to generate one or more types of high fidelity geometric proxies of the scene (block 604). By way of example but not limitation, the stream of calibrated point cloud reconstructions of the scene can be used to generate a stream of mesh models of the scene, where this mesh model generation can be performed using conventional methods such as Poisson, among others.
FIG. 7 illustrates an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of infrared images of the scene (hereafter simply referred to as different infrared image streams). As exemplified in FIG. 7, whenever the calibrated streams of sensor data include a plurality of different infrared image streams (block 700, Yes), the following actions occur. Any narrow baseline stereo pairs of sensors that exist in the arrangement of sensors and generate pairs of infrared image streams are identified (block 702). A first set of different depth map image streams is then created from the pairs of infrared image streams generated by the identified narrow baseline stereo pairs of sensors (block 704). Any wide baseline stereo pairs of sensors that exist in the arrangement of sensors and generate pairs of infrared image streams are then identified (block 706). A second set of different depth map image streams is then created from the pairs of infrared image streams generated by the identified wide baseline stereo pairs of sensors (block 708). The different depth map image streams in the first set and the second set are then merged into a stream of calibrated point cloud reconstructions of the scene (block 710). It is noted that these point cloud reconstructions are unordered and as such, overlaps may exist therein. Depending on the current pipeline conditions, the stream of calibrated point cloud reconstructions of the scene can then optionally be used to generate one or more types of high fidelity geometric proxies of the scene (block 712). By way of example but not limitation and as just described, the stream of calibrated point cloud reconstructions of the scene can be used to generate a stream of mesh models of the scene, although many applications will not need this level of fidelity.
FIG. 8 illustrates an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of color images of the scene (hereafter simply referred to as different color image streams). As exemplified in FIG. 8, whenever the calibrated streams of sensor data include a plurality of different color image streams (block 800, Yes), the following actions occur. Any narrow baseline stereo pairs of sensors that exist in the arrangement of sensors and generate pairs of color image streams are identified (block 802). A first set of different depth map image streams is then created from the pairs of color image streams generated by the identified narrow baseline stereo pairs of sensors (block 804). Any wide baseline stereo pairs of sensors that exist in the arrangement of sensors and generate pairs of color image streams are then identified (block 806). A second set of different depth map image streams is then created from the pairs of color image streams generated by the identified wide baseline stereo pairs of sensors (block 808). The different depth map image streams in the first set and the second set are then merged into a stream of calibrated point cloud reconstructions of the scene (block 810). It is noted that these point cloud reconstructions are unordered and as such, overlaps may exist therein. Depending on the current pipeline conditions, the stream of calibrated point cloud reconstructions of the scene can then optionally be used to generate one or more types of high fidelity geometric proxies of the scene (block 812). By way of example but not limitation and as just described, the stream of calibrated point cloud reconstructions of the scene can be used to generate a stream of mesh models of the scene.
It will be appreciated that depending on the particular arrangement of sensors that is used to capture the scene, a given sensor can be in a plurality of narrow baseline stereo pairs of sensors, and can also be in a plurality of wide baseline stereo pairs of sensors. This serves to maximize the number of different depth map image streams that are created, which in turn serves to maximize the precision of the scene proxy.

1.1.3.1 FVV Generation Examples

Referring again to FIG. 1, this section provides an overview description, in simplified form, of several additional implementations of the capture and processing stages 102 and 104 of the FVV processing pipeline 100. It will be appreciated that the implementations described in this section are merely exemplary. Many other implementations of the capture and processing stages 102 and 104 are also possible which use other types of sensor arrangements and generate other types of scene proxies.
In one implementation of the capture and processing stages of the FVV processing pipeline a circular arrangement of eight genlocked sensors is used to capture a scene which includes one or more human beings, where each of the sensors includes a combination of one infrared structured-light projector, two infrared video cameras, and one color camera. Accordingly, the sensors each generate a different stream of video data which includes both a stereo pair of infrared image streams and a color image stream. As described heretofore, the pair of infrared image streams and the color image stream generated by each sensor are first used to generate different depth map image streams. The different depth map image streams are then merged into a stream of calibrated point cloud reconstructions of the scene. These point cloud reconstructions are then used to generate a stream of mesh models of the scene. A conventional view-dependent texture mapping method which accurately represents specular textures such as skin is then used to extract texture data from the color image stream generated by each sensor and map this texture data to the stream of mesh models of the scene.
In another implementation of the capture and processing stages of the FVV processing pipeline four genlocked visible light video cameras are used to capture a scene which includes one or more human beings, where the cameras are evenly placed around the scene. Accordingly, the cameras each generate a different stream of video data which includes a color image stream. An existing 3D geometric model of a human body can be used in the scene proxy as follows. Conventional methods can be used to kinematically articulate the model over time in order to fit (i.e., match) the model to the streams of video data generated by the cameras. The kinematically articulated model can then be colored as follows. A conventional view-dependent texture mapping method can be used to extract texture data from the color image stream generated by each camera and map this texture data to the kinematically articulated model.
In another implementation of the capture and processing stages of the FVV processing pipeline three unsynchronized visible light video cameras are used to capture a soccer game, where each of the cameras is moving and is located far from the game (e.g., rather than the spatial location of each of the cameras being fixed to a specified arrangement, each of the cameras is hand held by a different user who is capturing the game while they freely move about). Accordingly, the cameras each generate a different stream of video data which includes a stream of color images of the game. Articulated billboards can be used to represent the moving players in the scene proxy of the game as follows. For each stream of video data, conventional methods can be used to generate a segmentation mask for each body part of each player in the stream. Conventional methods can then be used to generate an articulated billboard model of each of the moving players in the game from the appropriate segmentation masks. The articulated billboard model can then be colored as just described.

1.1.4 FVV Rendering Stage

This section provides a more detailed description of the rendering stage of the FVV processing pipeline. FIG. 9 illustrates an exemplary embodiment, in simplified form, of a process for rendering a FVV. As exemplified in FIG. 9, the process starts in block 900 with inputting scene proxies which geometrically describes the scene as a function of time. The scene proxies are then processed to generate a frame reflecting the current synthetic viewpoint of the scene, which maximizes the photo-realism thereof based upon a set of current pipeline conditions (block 902). These conditions can be in any one or more of the aforementioned stages of the FVV processing pipeline.
More particularly, FIG. 10 illustrates an exemplary embodiment, in simplified form, of a process for implementing the above-described rendering. As exemplified in FIG. 10, the process starts in block 1000 with monitoring and periodically analyzing the set of current pipeline conditions. The set of current pipeline conditions can include one or more conditions in the capture stage of the FVV processing pipeline such as the particular number of sensors that is being (or was) used to capture the scene, or the particular arrangement of these sensors that is being (or was) used, or one or more particular intrinsic characteristics of each of the sensors (e.g., the sensor type, among others), or one or more particular extrinsic characteristics of each of the sensors (e.g., the positioning of the sensor relative to the scene, and whether the sensor is static or moving, among others), or the complexity and composition of the scene, or whether the scene is relatively static or dynamic, or the like. The set of current pipeline conditions can also include one or more conditions in the processing stage of the pipeline such as the particular 3D reconstruction methods that are being (or were) used to generate the scene proxy, or the particular types of geometric proxy data that are in the scene proxy, or the like.
The set of current pipeline conditions can also include one or more conditions in the user viewing experience stage of the FVV processing pipeline such as the particular graphics processing capabilities [features] that are available in the computing device hardware which is being used, or the particular type of display device the rendered FVV frames are being displayed on, or the particular characteristics of the display device (described heretofore), or the particular number of degrees of viewpoint navigation freedom that are being provided to the end user, or whether or not the end user's client computing device includes a natural user interface (and if so, the particular natural user interface modalities that are anticipated to be used by the end user), or the like. The set of current pipeline conditions can also include information which is generated by the end user and provided to the user viewing experience stage that specifies desired changes to (i.e., controls) the current synthetic viewpoint of the scene. Such information can include viewpoint navigation information which is being input by this stage based upon the FVV navigation that is being performed by the end user, or temporal navigation information which may also be input to this stage based upon this FVV navigation. The set of current pipeline conditions can also include the particular type of FVV that is being processed in the pipeline.
Referring again to FIG. 10, after the analysis of the set of current pipeline conditions has been completed (block 1000), the results of this analysis are then used to select one or more different image-based rendering methods which are matched to the current pipeline conditions (block 1002). The selected image-based rendering methods and the results of the period analysis are then used to generate a frame reflecting the current synthetic viewpoint of the scene (block 1004). The actions of blocks 1000, 1002 and 1004 are repeated for the duration of the scene (block 1006, No). As such, it will be appreciated that the one or more image-based rendering methods which are used and frames that are generated change over time based upon changes in the current pipeline conditions. As described heretofore, the current pipeline conditions can be analyzed using different periodicities.
It will thus be appreciated that the exemplary pipeline described herein can use a wide variety of image-based rendering methods in various combinations, where the particular types of image-based rendering methods that are being used depend upon various current conditions in the FVV processing pipeline. The image-based rendering methods that are employed by the pipeline techniques described herein can render novel views (i.e., synthetic viewpoints) of the scene directly from a collection of images in the scene proxy without having to know the scene geometry. An overview exemplary image-based rendering methods which can be employed by the pipeline are provided hereafter.
The pipeline supports using any type of display device to view the FVV including, but not limited to, the very small form factor display devices used on conventional smart phones and other types of mobile devices, the small form factor display devices used on conventional tablet computers and netbook computers, the display devices used on conventional laptop computers and personal computers, conventional televisions and 3D televisions, conventional autostereoscopic 3D display devices, conventional head-mounted transparent display devices, and conventional wearable heads-up display devices such as those that are used in virtual reality applications. In a situation where the end user is using an autostereoscopic 3D display device to view the FVV, then the rendering stage of the FVV processing pipeline will simultaneously generate both left and right current synthetic viewpoints of the scene at an appropriate aspect ratio and resolution in order to create a stereoscopic effect for the end user. In another situation where the end user is using a conventional television to view the FVV, then the rendering stage will generate just a single current synthetic viewpoint. In yet another situation where the end user is viewing the FVV in an augmented reality context, (e.g., in a situation where the end user is wearing a head-mounted transparent display), then the rendering stage may generate a current synthetic viewpoint having just the foreground elements of the captured scene, thus enabling objects to be embedded in a natural environment.
The pipeline also supports using any type of user interface modality to control the current viewpoint while viewing the FVV including, but not limited to, conventional keyboards, conventional pointing devices (such as a mouse, or a graphics tablet, or the like), and conventional natural user interface modalities (such as a touch-sensitive display screen, or the head tracking functionality that is integrated into wearable heads-up display devices, or a motion and location sensing device (such as the Microsoft Kinect™, among others). It will be appreciated that if the end user is (or will be) using of one or more natural user interface modalities while they are viewing the FVV, this can influence the spatiotemporal navigation capabilities that are provided to the end user. In other words, the FVV processing pipeline can process the streams of sensor data differently in order to enable different end user viewing experiences based on the particular type(s) of user interface modality that is anticipated to be used by the end user. By way of example but not limitation, in a situation where a given end user is using the wearable heads-up display device to view and navigate the FVV, then all six degrees of viewpoint navigation freedom could be provided to the end user. In the bidirectional live FVV implementation of the pipeline technique embodiments, if the end user at each physical location that is participating in a given videoconference/telepresence session is using the wearable heads-up display device to view and navigate the FVV, then parallax functionality can be implemented in order to provide each end user with an optimally realistic viewing experience when they control/change their viewpoint of the FVV using head movements; the pipeline can also provide for corrected conversational geometry between two end users, thus providing the appearance that both end users are looking directly at each other. In another situation where a given end user is using the motion and location sensing device navigate the FVV, then the rendering stage can optimize the current synthetic viewpoint that is being displayed based on the end user's current spatial location in front of their display device. In this way, the end user's current spatial location can be mapped to the 3D geometry within the FVV.
FIG. 11 illustrates the various degrees of viewpoint navigation freedom that can be supported by the pipeline techniques describe herein. As described heretofore, the pipeline generally supports spatiotemporal (i.e., space-time) navigation of the FVV. More particularly, the recorded FVV, unidirectional live FVV, and bidirectional live FVV implementations described herein can each support spatial viewpoint navigation of the FVV having as many as six degrees of freedom, which can be appropriate when the end user is viewing and navigating an FVV that includes high fidelity geometric information. As exemplified in FIG. 11, these six degrees of freedom include viewpoint navigation along the x axis, viewpoint navigation rotationally about the x axis (8×), viewpoint navigation along the y axis, viewpoint navigation rotationally about the y axis (θy), viewpoint navigation along the z axis, and viewpoint navigation rotationally about the z axis (θz). The recorded FVV, unidirectional live FVV, and bidirectional live FVV implementations can also each support spatial viewpoint navigation of the FVV having just one degree of viewpoint navigation freedom. The recorded FVV implementation can also support temporal navigation of the FVV.
In some implementations of the pipeline, such as the recorded FVV implementation described herein, a producer or editor of the FVV may want to specify the particular types of viewpoint navigation that is possible at different times during the FVV. By way of example but not limitation, in one scene a movie director may want to confine the end user's viewpoint navigation to a limited area of the scene or a specific axis, but in another scene the director may want to allow the end user to freely navigate their viewpoint throughout the entire area of the scene.

1.1.4.1 Image-Based Rendering Methods

As described heretofore, the current synthetic viewpoint of the scene is generated using one or more image-based rendering methods which are selected based upon a periodic analysis of the aforementioned set of current pipeline conditions. Accordingly, the particular image-based rendering methods that are used can change over time based upon changes in the current pipeline conditions. It will thus be appreciated that in one situation where the scene has a low degree of complexity and the arrangement of sensors which is being (or was) used to capture the scene are located close to the scene, just a single image-based rendering method may be used to generate the current synthetic viewpoint of the scene. In another situation where the scene has a high degree of complexity and the arrangement of sensors which is being (or was) used to capture the scene are located far from the scene, a plurality of image-based rendering methods may be used to generate the current synthetic viewpoint of the scene depending on the location of the current viewpoint relative to the scene and the particular types of geometric proxy data that are in the scene proxy.
FIG. 12 illustrates an exemplary embodiment, in simplified form, of a continuum of the various image-based rendering methods which can be employed by the pipeline technique embodiments described herein. As exemplified in FIG. 12, for didactic purposes these various image-based rendering methods can be classified into three categories according to the amount and type of scene geometry information that is included in the scene proxy and thus is available to be used in the rendering stage, namely rendering with without scene geometry 1206 (i.e., the scene geometry is unknown), rendering with implicit scene geometry 1204 (i.e., correspondence), and rendering with explicit scene geometry 1202 (which can be either approximate or accurate). These categories 1202, 1204 and 1206 are to be viewed as a continuum 1200 rather than strict and discrete categories since it will be appreciated that certain of the image-based rendering methods defy strict categorization.
As also exemplified in FIG. 12 a trade-off exists between the amount and type of scene geometry information that is available to be used in the rendering stage, and the number of images that are needed to be available in the scene proxy in order to generate synthetic viewpoints of the scene which are photo-realistic. Generally speaking, the higher the sensor density in the arrangement of sensors that is being used to capture the scene (i.e., the larger the number of sensors that is used in the arrangement), the larger the number of images that is available in the scene proxy, and thus the less scene geometry information that is needed to be available in the scene proxy in order to generate synthetic viewpoints of the scene which are photo-realistic. However, it is noted that having less scene geometry information in the scene proxy will generally decrease the end user's options for navigating the current synthetic viewpoint of the scene (i.e., the synthetic viewpoints will generally be limited to positions between sensors or near sensors. Correspondingly, the lower the sensor density (i.e., the smaller the number of sensors that is used in the arrangement), the smaller the number of images that is available in the scene proxy, and thus the more scene geometry information that is needed to be available in the scene proxy in order to generate synthetic viewpoints of the scene which are photo-realistic. However, it is noted that having more scene geometry information in the scene proxy will generally increase the end user's options for navigating the current synthetic viewpoint of the scene (i.e., the synthetic viewpoints can be navigated to positions which are far away from the real sensor viewpoints).
On the left side 1206 of the continuum 1200 exemplified in FIG. 12 the scene proxy includes a large number of images but does not include any scene geometry or correspondence information. In this situation a conventional light field method, or a conventional lumigraph method, or a conventional concentric mosaics method, among others, can be used to process the scene proxy in order to generate the current synthetic viewpoint of the scene. As is appreciated in the art of image-based rendering, each of these methods relies on the characterization of the conventional plenoptic function, and constructs a continuous representation of the plenoptic function from the images in the scene proxy. The light field method is generally applicable when the images of the scene are uniformly captured. The light field method generates new images of the scene by appropriately filtering and interpolating the images in the scene proxy. The lumigraph method is similar to the light field method except that the lumigraph method is generally applicable when the images of the scene are not uniformly captured. The lumigraph method enhances the rendering performance by applying approximated geometry to compensate for this non-uniform capture. Unlike the light field and lumigraph methods which are applicable when the arrangement of sensors is a 2D grid, the concentric mosaics method is applicable when the arrangement of sensors is circular. Conventional image mosaicing methods can also be used to construct a complete plenoptic function at a fixed viewpoint from an incomplete set of images of the scene.
In the middle 1204 of the continuum 1200 exemplified in FIG. 12 the scene proxy does not include explicit scene geometry information, but rather it includes implicit scene geometry information in the form of feature (e.g., point) correspondences between images, where these correspondences can be computed using conventional computer vision methods. In this situation various conventional transfer methods (such as a conventional view interpolation method, or a conventional view morphing method, among others) can be used to process the scene proxy in order to generate the current synthetic viewpoint of the scene. As is appreciated in the art of image-based rendering, such transfer methods are characterized by the use of a relatively small number of images with the application of geometric constraints (which are either recovered or known a priori) to project image pixels appropriately at a given synthetic viewpoint. These geometric constraints can be in the form of known depth values at each pixel, or epipolar constraints between stereo pairs of images, or trifocal/tri-linear tensors that link correspondences between triplets of images. The view interpolation method generates synthetic viewpoints of the scene by interpolating optical flow between corresponding points. The view morphing method generates synthetic viewpoints that reside on a line which links the optical centers of two different sensors based on point correspondences.
On the right side 1202 of the continuum 1200 exemplified in FIG. 12 the scene proxy includes explicit and accurate scene geometry information and a small number of images, where this geometry information can be in form of either depth along known lines-of-sight, or 3D coordinates, among other things. In this situation conventional 3D warping methods, or a conventional layered depth images method, or a conventional layered depth images tree method, or a conventional view-dependent texture mapping method, or a conventional view-dependent geometry method, among others, can be used to process the scene proxy in order to generate the current synthetic viewpoint of the scene. As is appreciated in the art of image-based rendering, the 3D warping methods, or the layered depth images method, or the layered depth images tree method can be used when the scene proxy includes both depth map images and color (or monochrome) images of the scene. When the scene proxy includes depth information for all the points in an image, the 3D warping methods can be used to render the image from any nearby point of view by projecting the pixels of the image to their proper 3D locations and then re-projecting them onto a new picture. The rendering speed of such 3D warping methods can be increased by using conventional relief texture methods which factor the warping process into a relatively simple pre-warping operation and a conventional texture mapping operation (which may be performed by conventional graphics processing hardware). It is noted that the 3D warping methods can be applied to both traditional perspective images as well as multi-perspective images. The view-dependent geometry method was first used in the context of 3D cartoons and trades off geometry and images, and may be used to represent the current synthetic viewpoint of the scene more compactly. A conventional texture-mapped models method can also be used to generate the current synthetic viewpoint of the scene.

1.1.5 FVV User Viewing Experience Stage and Interactive FVV Presentation

This section provides a more detailed description of the user viewing experience stage of the FVV processing pipeline, and the presentation of a FVV to one or more end users. As described previously, each end user interactively navigates their viewpoint of the scene via their client computing device, and each time an end user chooses a different viewpoint, this new viewpoint is provided to the user viewing experience stage by the user's client computing device. To this end, each end user has a FVV player operating on their client computing device. The FVV player facilitates the display of FVV related items (e.g., FVV frames or user interface screens), accepts end user inputs, and causes the client computing device to communicate with the FVV user experience stage. For example, as outlined in the exemplary embodiment FIGS. 13A-B, once an end user installs the FVV player on his or her client computing device (via conventional methods), the player can be initiated by the end user and requested to display a user interface screen that allows the user to select a FVV available for playing (block 1300). As noted previously, this can be a live FVV or a previously recorded FVV. The end user FVV selection is then input (block 1302), and transmitted via the client computing device to the server upon which the user experience stage is operating (block 1304). The user experience stage receives the user selection (block 1306), and in response, initiates the FVV pipeline (more particularly, the rendering stage) to produce the selected FVV (block 1308), and in block 1310 instructs the client device to instantiate the end user controls in the FVV player appropriate to the FVV type. The client computing device receives the instruction (block 1312), and causes a user interface to be displayed that allows the user to interactively control the viewpoint of the FVV scene (block 1314). As indicated previously, this viewpoint control can be spatial, temporal, or both, depending on the FVV selected. Meanwhile, the rendering stage of the FVV pipeline begins rendering frames of the selected FVV (block 1316) and providing them for display via the user experience stage and the end user's client device (block 1318). It is noted that the initial viewpoint of the FVV scene depicted in the initial frames rendered can be a default viewpoint assigned to the FVV if the end user has not yet specified a viewpoint. As the FVV frames are provided, the end user's client computing device receives them (block 1320) and in block 1322 displays them on a resident display device (such as one described previously). This procedure of rendering, providing and displaying of FVV frames at the current viewpoint is repeated at an appropriate frame rate for the FVV, while at the same time the client computing device monitors end user inputs to the FVV player to determine if an end user viewpoint navigation input has been received (block 1324). If such an input is detected, the client computing device transmits it to the user experience stage of the FVV pipeline (block 1326), which receives it and forwards the new viewpoint to the rendering stage (block 1328). The rendering stage then renders FVV frames depicting the scene from the new (now current) viewpoint (block 1330) and provides them for display as described previously (block 1332). As the FVV frames are provided, the end user's client computing device receives them (block 1334) and in block 1336 displays them on a resident display device. This monitoring and rendering procedure is then repeated for the duration of the FVV.
It is noted that in situations where a previously recorded FVV is being viewed, as indicated above an end user can temporally control the playback of the FVV, and based on this temporal control the rendering stage will provide FVV frames starting with the frame that corresponds the last user-specified temporal location in the FVV. More particularly, referring to FIG. 14, in one embodiment, the aforementioned scene is first captured by a server using an arrangement of sensors (block 1400). These streams of sensor data are input and calibrated (block 1402), and then scene proxies are generated from the calibrated streams of sensor data (block 1404). The scene proxies are stored as they are generated (block 1406). Meanwhile, a client computing device monitors navigational inputs from an end user (block 1408). A synthetic viewpoint of the scene is input by the end user and the client computing device transmits the current viewpoint input to the server via the data communication network (block 1410). Additionally, a temporal navigation instruction is input by the end user and the client computing device transmits it to the server as well (block 1412). This temporal navigation input represents an instruction to provide FVV frames from a user-specified temporal location in the FVV. The current synthetic viewpoint of the scene and the temporal navigation instruction are received from the client computing device (block 1414). A sequence of frames is then generated using the scene proxies (block 1416). Each frame of the sequence depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene, the first of which corresponds to the last user-specified temporal location in the FVV. The generated frames are then transmitted to the client computing device via the data communication network (block 1418). The client computing device receives the frames (block 1420), and displays them in a conventional manner to the end user (block 1422).
In another embodiment, the FVV can be played in reverse, thus rewinding the FVV while still allowing the end user to watch. More particularly, referring to FIG. 15, in one embodiment, the aforementioned scene is captured by a server using an arrangement of sensors (block 1500). These streams of sensor data are input and calibrated (block 1502), and then scene proxies are generated from the calibrated streams of sensor data (block 1504). The scene proxies are stored as they are generated (block 1506). Meanwhile, a client computing device monitors navigational inputs from an end user (block 1508). A synthetic viewpoint of the scene is input by the end user and the client computing device transmits the current viewpoint input to the server via the data communication network (block 1510). Additionally, a reverse-action temporal navigation instruction is input by the end user and the client computing device transmits it to the server as well (block 1512). This reverse-action temporal navigation input represents an instruction to provide FVV frames in reverse order from a specified temporal location in the FVV, thereby rewinding the FVV. The current synthetic viewpoint of the scene and the reverse-action temporal navigation instruction are received from the client computing device (block 1514). A sequence of frames is then generated using the scene proxies (block 1516). Each frame of the sequence depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene, and are generated in reverse order, the first of which corresponds to the last user-specified temporal location in the FVV. The generated frames are then transmitted to the client computing device via the data communication network (block 1518). The client computing device receives the frames (block 1520), and displays them in a conventional manner to the end user (block 1522).
In yet another embodiment, the FVV can be paused and restarted by the end user. More particularly, referring to FIG. 16, in one embodiment, as the aforementioned FVV frames are received by the client computing device and displayed to the end user, the client computing device monitors inputs from the end user (block 1600). A pause instruction is input by the end user and the client computing device transmits it to the server via the data communication network (block 1602). The pause instruction is received from the client computing device (block 1604). The server then suspends the generation and transmission of FVV frames to the client computing device (block 1606). While the FVV is paused, the client computing device continues to monitor inputs from the end user (block 1608). A restart instruction is input by the end user and the client computing device transmits it to the server via the data communication network (block 1610). The restart instruction is received from the client computing device (block 1612). The server then restarts the generation and transmission of FVV frames to the client computing device (block 1614). The client computing device receives the frames (block 1616), and displays them in a conventional manner to the end user (block 1618).

2.0 Additional Embodiments

While the foregoing cloud based FVV streaming technique embodiments have been described by specific reference to embodiments thereof, it is understood that variations and modifications thereof can be made without departing from the true spirit and scope of the pipeline technique. For example, additional embodiments can be designed to reduce latency times and employed when latency issues are a concern.
By way of example but not limitation, in one such additional embodiment, each frame transmitted to a client computer is also accompanied with at least some of the scene proxies used by the renderer to generate the frame. This allows the client device to locally generate a new frame of the depicted scene from a different viewpoint in the same manner the renderer produces frames when a new viewpoint is requested by the end user (as described previously). More particularly, whenever a same-frame end user viewpoint navigation input is received via the aforementioned FVV control user interface which represents an instruction to view a scene depicted in the last-displayed FVV frame from a different viewpoint, the client computing device generates a new FVV frame using the scene proxy or proxies received with the last-displayed frame, and displays the new FVV frame on the aforementioned display device. This new FVV frame depicts the scene depicted in the last-displayed FVV frame from a viewpoint specified in the same-frame end user viewpoint navigation input.
In another additional embodiment, the frame transmitted to a client computing device would depict all or a larger portion of the captured scene, than the display device associated with the client computing device is capable of displaying. Thus, only a portion of the received frame could be displayed at one time. This allows an end user to translate through the depicted scene without having to request a new frame from the FVV pipeline. More particularly, whenever a “same-frame” end user viewpoint navigation input is received via the FVV control user interface which represents an instruction to view a portion of the scene depicted in the last-received FVV frame that was not shown in the last-displayed portion of the frame, at least the portion of the scene depicted in the last-received FVV frame specified in the same-frame end user viewpoint navigation input is displayed on the display device.
Still another additional embodiment involves the FVV pipeline, and more particularly, the rendering stage predicting the next new viewpoint to be requested. For example, this can be accomplished based on past viewpoint change requests received from a end user. The rendering stage then renders and stores a new frame (or a sequence of frames) from the predicted viewpoint, and provides it the client computing device of the end-user if that end user requests the predicted viewpoint. It is further noted that the rendering stage could render multiple frames based on multiple predictions of what viewpoint the end user might request next. Then, if the end user's next viewpoint request matches one of the rendered frames, that frame is sent to the end user's client computing device. More particularly, referring to FIGS. 17A-B, in one embodiment, the foregoing procedure is accomplished as follows. First, the server predicts one or more synthetic viewpoints of the scene that may be received from a client computing device in the future (block 1700). A previously unselected one of the predicted synthetic viewpoints of the scene is then selected (block 1702), and one or more frames are generated using the aforementioned stored scene proxies which depict at least a portion of the scene as viewed from the selected predicted synthetic viewpoint of the scene (block 1704). The generated frame or frames are then stored (block 1706). It is then determined if all the predicted synthetic viewpoints have been selected (block 1708). If not, blocks 1702 through 1708 are repeated. However, if all the predicted synthetic viewpoints have been selected and frames therefor generated and stored, the incoming messages from the client computing device are monitored (block 1710), and when a message is received it is determined if the message includes a current synthetic viewpoint of the scene that matches one of the predicted synthetic viewpoints (block 1712). If the message includes a current synthetic viewpoint of the scene that matches one of the predicted synthetic viewpoints, then each frame generated based on the matched predicted synthetic viewpoint of the scene is transmitted to the client computing device via the data communication network for display to the end user of the client computing device (block 1714). Then, the previously described process (as described in connection with FIG. 2) resumes. This is also the case if the message does not include a current synthetic viewpoint that matches one of the predicted synthetic viewpoints. More particularly, a sequence of frames is generated using the scene proxies where each frame depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene (block 1716), and each frame is transmitted to the client computing device via the data communication network for display to the end user (block 1718).
It is also noted that any or all of the aforementioned embodiments can be used in any combination desired to form additional hybrid embodiments. Although the pipeline technique embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described heretofore. Rather, the specific features and acts described heretofore are disclosed as example forms of implementing the claims.

3.0 Computing Environment

The cloud based FVV streaming technique embodiments described herein are operational within numerous types of general purpose or special purpose computing system environments or configurations. FIG. 18 illustrates a simplified example of a general-purpose computer system on which various embodiments and elements of the pipeline technique, as described herein, may be implemented. It is noted that any boxes that are represented by broken or dashed lines in FIG. 18 represent alternate embodiments of the simplified computing device, and that any or all of these alternate embodiments, as described below, may be used in combination with other alternate embodiments that are described throughout this document.
For example, FIG. 18 shows a general system diagram showing a simplified computing device 10. Such computing devices can be typically be found in devices having at least some minimum computational capability, including, but not limited to, personal computers (PCs), server computers, handheld computing devices, laptop or mobile computers, communications devices such as cell phones and personal digital assistants (PDAs), multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and audio or video media players.
To allow a device to implement the cloud based FVV streaming technique embodiments described herein, the device should have a sufficient computational capability and system memory to enable basic computational operations. In particular, as illustrated by FIG. 18, the computational capability is generally illustrated by one or more processing unit(s) 12, and may also include one or more graphics processing units (GPUs) 14, either or both in communication with system memory 16. Note that that the processing unit(s) 12 may be specialized microprocessors (such as a digital signal processor (DSP), a very long instruction word (VLIW) processor, an FPGA or other micro-controller) or can be conventional central processing units (CPUs) having one or more processing cores including, but not limited to, specialized GPU-based cores in a multi-core CPU.
In addition, the simplified computing device 10 of FIG. 18 may also include other components, such as, for example, a communications interface 18. The simplified computing device 10 of FIG. 18 may also include one or more conventional computer input devices 20 (e.g., pointing devices, keyboards, audio input/capture devices, video input/capture devices, haptic input devices, devices for receiving wired or wireless data transmissions, and the like). The simplified computing device 10 of FIG. 18 may also include other components, such as, for example, display device(s) 24, and one or more conventional computer output devices 22 (e.g., audio output devices, video output devices, devices for transmitting wired or wireless data transmissions, and the like). Exemplary types of input devices (herein also referred to as user interface modalities) and display devices that are operable with the pipeline technique embodiments described herein have been described heretofore. Note that typical communications interfaces 18, additional types of input and output devices 20 and 22, and storage devices 26 for general-purpose computers are well known to those skilled in the art, and will not be described in detail herein.
The simplified computing device 10 of FIG. 18 may also include a variety of computer readable media. Computer readable media can be any available media that can be accessed by the computer 10 via storage devices 26, and includes both volatile and nonvolatile media that is either removable 28 and/or non-removable 30, for storage of information such as computer-readable or computer-executable instructions, data structures, program modules, or other data. By way of example but not limitation, computer readable media may include computer storage media and communication media. Computer storage media includes, but is not limited to, computer or machine readable media or storage devices such as digital versatile disks (DVDs), compact discs (CDs), floppy disks, tape drives, hard drives, optical drives, solid state memory devices, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other device which can be used to store the desired information and which can be accessed by one or more computing devices.
Storage of information such as computer-readable or computer-executable instructions, data structures, program modules, and the like, can also be accomplished by using any of a variety of the aforementioned communication media to encode one or more modulated data signals or carrier waves, or other transport mechanisms or communications protocols, and includes any wired or wireless information delivery mechanism. Note that the terms “modulated data signal” or “carrier wave” generally refer a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wired connection carrying one or more modulated data signals, and wireless media such as acoustic, radio frequency (RF), infrared, laser, and other wireless media for transmitting and/or receiving one or more modulated data signals or carrier waves. Combinations of the any of the above should also be included within the scope of communication media.
Furthermore, software, programs, and/or computer program products embodying the some or all of the various embodiments of the pipeline technique described herein, or portions thereof, may be stored, received, transmitted, or read from any desired combination of computer or machine readable media or storage devices and communication media in the form of computer executable instructions or other data structures.
Finally, the cloud based FVV streaming technique embodiments described herein may be further described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The cloud based FVV streaming technique embodiments may also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices, or within a cloud of one or more devices, that are linked through one or more communications networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices. Additionally, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor.

Claims

Wherefore, what is claimed is:

1. A computer-implemented process for generating a free viewpoint video (FVV) of a scene, comprising:

using one or more computing devices to perform the following process actions:

capturing the scene using an arrangement of sensors, said arrangement comprises a plurality of sensors that generate a plurality of streams of sensor data each of which represents the scene from a different geometric perspective;

inputting and calibrating the streams of sensor data;

generating scene proxies from the calibrated streams of sensor data, which geometrically describe the scene as a function of time;

receiving a current synthetic viewpoint of the scene from a client computing device via a data communication network, said current synthetic viewpoint having been selected by an end user of the client computing device;

generating a sequence of frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene; and

transmitting each frame generated to the client computing device via the data communication network for display to the end user of the client computing device.

2. The process of claim 1, further comprising, prior to performing the process action of receiving a current synthetic viewpoint of the scene from the client computing device, performing the actions of:

generating a sequence of frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from a prescribed default viewpoint of the scene; and

3. The process of claim 1, wherein the process action of generating a scene proxy comprises the actions of:

monitoring and periodically analyzing current conditions related to at least one of capturing streams of sensor data, or calibrating the streams of sensor data, or generating scene proxies, or generating a sequence of frames, or transmitting each frame generated; and

each time the current conditions are analyzed,

using results of the periodic analysis to select one or more different 3D reconstruction methods which are matched to the current conditions,

using each selected 3D reconstruction method to generate a 3D reconstruction of at least part of the scene from the calibrated streams of sensor data, and

using the 3D reconstructions and the results of the periodic analysis to generate scene proxies.

4. The process of claim 1, wherein each scene proxy comprises one or more types of geometric proxy data, and wherein the process action of generating scene proxies, comprises an action of, for each scene proxy generated, matching the scene proxy to a set of current conditions related to at least one of capturing streams of sensor data or calibrating the streams of sensor data, so as to maximize the photo-realism of frames generated from the scene proxy.

5. The process of claim 1, wherein the process action of generating a sequence of frames, comprises the actions of,

each time the current conditions are analyzed,

using results of the periodic analysis to select one or more different image-based rendering methods which are matched to the current conditions, and

using each selected image-based rendering method and the results of the period analysis to generate the sequence of frames.

6. The process of claim 1, further comprising an action of, for each frame transmitted to the client computing device, also transmitting at least some of the scene proxies used to generate the frame.

7. The process of claim 1, wherein the process action of generating a sequence of frames, comprises generating each frame so as to depict at least a portion of the scene as viewed from the current synthetic viewpoint of the scene that is larger than a display device associated with the client computing device is capable of displaying.

8. The process of claim 1, further comprising the actions of:

predicting one or more synthetic viewpoints of the scene that may be received from the client computing device in the future;

for each predicted synthetic viewpoint of the scene,

generating one or more frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from the predicted synthetic viewpoint of the scene, and

storing each frame generated based on the predicted synthetic viewpoint of the scene; and

whenever a current synthetic viewpoint of the scene is received from the client computing device that matches one of the predicted synthetic viewpoints, prior to performing the process action of generating a sequence of frames using the scene proxies, transmitting each frame generated based on the matched predicted synthetic viewpoint of the scene to the client computing device via the data communication network for display to the end user of the client computing device.

9. The process of claim 1, further comprising:

prior to performing the process actions of generating and transmitting a sequence of frames, performing the process actions of,

storing each scene proxy generated, and

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to provide FVV frames from a specified temporal location in the FVV; and wherein

the process action of generating a sequence of frames further comprises using the stored scene proxies to generate frames, the first of which corresponds to the last user-specified temporal location in the FVV.

10. The process of claim 1, further comprising:

performing prior to performing the process actions of generating and transmitting a sequence of frames, the process actions of,

storing each scene proxy generated, and

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to provide FVV frames in reverse order from a specified temporal location in the FVV thereby rewinding the FVV; and wherein

the process action of generating a sequence of frames comprises using the stored scene proxies to generate frames in reverse order, the first of which corresponds to the last user-specified temporal location in the FVV.

11. The process of claim 1, further comprising the process actions of:

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to pause the FVV; and

suspending the generation and transmission of FVV frames to the client computing device.

12. The process of claim 11, further comprising the process actions of:

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to restart the paused FVV; and

restarting the generation and transmission of FVV frames to the client computing device.

13. A system for generating a free viewpoint video (FVV) of a scene, comprising:

an arrangement of sensors used to capture video of the scene, or audio, or both, each of said sensors producing a stream of sensor data representing the scene from a different geometric perspective;

at least one general purpose computing device; and

a computer program comprising program modules executed by the computing device or devices, wherein the computing device or devices are directed by the program modules of the computer program to,

input and calibrate the streams of sensor data produced by the arrangement of sensors,

generate scene proxies from the calibrated streams of sensor data, which geometrically describe the scene as a function of time,

store the scene proxies as they are generated,

generate a sequence of frames using the stored scene proxies, each frame of which depicts at least a portion of the scene as viewed from a current synthetic viewpoint of the scene, and

transmit each frame generated to a client computing device via a data communication network for display to the end user of the client computing device.

14. The system of claim 13, further comprising a program module for receiving current synthetic viewpoints of the scene from a client computing device via a data communication network, each of said current synthetic viewpoints having been selected by an end user of the client computing device, and wherein said program module for generating a sequence of frames using the stored scene proxies, employs each current synthetic viewpoint received to generate a sequence of frames, each frame of which depicts at least a portion of the scene as viewed from the last-received current synthetic viewpoint.

15. The system of claim 14, wherein prior to receiving a first current synthetic viewpoint of the scene from a client computing device, said program module for generating a sequence of frames using the stored scene proxies, employs a prescribed default synthetic viewpoint as the current synthetic viewpoint to generate a sequence of frames of the scene.

16. The system of claim 13, wherein prior to executing the program module for transmitting each frame generated, executing additional program modules for:

storing each frame generated; and

the program module for transmitting each frame generated to the client computing device, comprises transmitting frames, the first of which corresponds to a frame assigned to the last user-specified temporal location in the FVV.

17. A computer-implemented process for playing a free viewpoint video (FVV) of a scene, comprising:

using a client computing device to perform the following process actions:

inputting a request from an end user to display a FVV selection user interface screen that allows the end user to select a FVV available for playing;

displaying the FVV selection user interface screen on a display device;

inputting an end user FVV selection;

transmitting the end user FVV selection to a server via a data communication network;

receiving an instruction from the server via the data communication network that instructs the client computing device to instantiate end user controls appropriate for the type of FVV selected;

displaying a FVV control user interface on said display device;

monitoring for end user inputs via the FVV control user interface;

whenever an end user viewpoint navigation input is received via the FVV control user interface, transmitting the input to the server via the data communication network;

receiving FVV frames from the server via the data communication network, each FVV frame depicting at least a portion of the scene as it would be viewed from either an initial viewpoint if the end user has not yet input a viewpoint navigation input via the FVV control user interface or the last viewpoint the end user input using a viewpoint navigation input via the FVV control user interface; and

displaying each FVV frame on said display device as it is received.

18. The process of claim 17, further comprising process actions of:

whenever an end user temporal navigation input is received via the FVV control user interface, transmitting the input to the server via the data communication network, said temporal navigation input representing an instruction to provide FVV frames from a user-specified temporal location in the FVV; and

receiving a sequence of FVV frames from the server via the data communication network, the first of which corresponds to the last user-specified temporal location in the FVV.

19. The process of claim 17, further comprising the process actions of:

receiving with each FVV frame received from the server, one or more scene proxies which were used by the server to generate the received FVV frame; and

whenever a same-frame end user viewpoint navigation input is received via the FVV control user interface which represents an instruction to view a scene depicted in the last-displayed FVV frame from a different viewpoint,

generating a new FVV frame using the scene proxy or proxies received with the last-displayed frame, said new FVV frame depicting the scene depicted in the last-displayed FVV frame from a viewpoint specified in the same-frame end user viewpoint navigation input, and

displaying the new FVV frame on said display device.

20. The process of claim 17, wherein the FVV frames received from the server depict at least a portion of the scene as viewed from the current synthetic viewpoint of the scene that is larger than said display device is capable of displaying, and only a portion of each FVV frame is displayed on said display device, the process further comprising a process actions of, whenever a same-frame end user viewpoint navigation input is received via the FVV control user interface which represents an instruction to view a portion of the scene depicted in the last-received FVV frame that was not shown in the last-displayed portion of the frame, displaying at least the portion of the scene depicted in the last-received FVV frame specified in the same-frame end user viewpoint navigation input on said display device.