Completed Projects

These projects represent work in Perceiving Systems between Jan 2011 and the present that has been superseded by new work or that we are no longer pursuing.

Projects

Human Pose, Shape and Action

Human pose estimation, 3D mesh registration and action recognition techniques have made significant progress during the last years. However, most existing datasets to evaluate them are inadequate for capturing the... Read more

3D Pose from Images

The estimation of 3D human pose from 2D images is inherently ambiguous. To that end, we develop inference methods and human pose models that enable prediction of 3D pose from images. Learned models of human pose rely on... Read more

2D Pose from Images

Estimating 2D human pose is hard because people appear in a wide range of poses and have varying body shape. They wear varied clothing and the articulation results in significant self occlusion. We have developed several state-of-the-art... Read more

Beyond Motion Capture

Accurately capturing human body shape and motion is important for many applications in computer vision and graphics. Traditional motion capture (mocap) focuses on extracting a skeleton from a sparse set of markers. Our work... Read more

Action and Behavior

Human behavior can be described at multiple levels. At the lowest level, we observe the 3D pose of the body over time. Poses can be organized into primitives that capture coordinated activity of different body parts. These further form... Read more

Body Perception

Body representation is an essential part of a person’s self-concept and also shapes how we see the world. A disturbed body representation also plays a role in clinical conditions such as eating disorders or stroke. So far, a major... Read more

Body Applications

Accurate models of human body shape and appearance are powerful tools to tackle a variety of problems in medicine, ergonomics, fashion, fitness, etc. In [

], we propose a solution for extracting anthropometric or tailoring... Read more

Pose and Motion Priors

A prior over human pose is important for many human tracking and pose estimation problems. We introduce a sparse Bayesian network model of human pose that is non-parametric with respect to the estimation of both its graph... Read more

Clothing Models (2011-2015)

Clothed virtual characters in varied sizes and shapes are necessary for film, gaming, and on-line fashion applications. Dressing such characters is a significant bottleneck, requiring manual effort to design clothing,... Read more

Reflectance Filtering

This is the project page for the paper "Reflectance Adaptive Filtering Improves Intrinsic Image Estimation" [

], which appears at CVPR 2017 and can be found at https://arxiv.org/abs/1612.05062. Code is available at ... Read more

Learning on Manifolds

Statistics is traditionally concerned with data in a Euclidean space, relying on the linear structure and the distances associated with such a space; this renders it inappropriate for nonlinear spaces. Statistics can, however, be... Read more

Markerless Animal Motion Capture

Experiments in motor neurophysiology often involve animals performing repeated actions. Stereotyped and practiced actions facilitate data analysis by allowing the experimenter to average neural firing activity across multiple... Read more

Multi-Camera Capture

While multi-camera video data facilitates markerless motion capture, many challenges remain. We formulate the problem of 3D human pose estimation and tracking as inference in a graphical model [

]. The body is... Read more

2D Pose from Optical Flow

Much of the work on human pose estimation focuses on still images. We argue that there is much to be gained by looking at video sequences and, specifically, using optical flow. Flow tells us what goes with what over time.... Read more

Body Perception

We create virtual avatars from full body 3D scans and then manipulate body shape, pose, and appearance to create realistic stimuli for the study of the human perception of body shape. We created... Read more

Neural Prosthetics and Decoding

We use motion capture together with electrode arrays, implanted in the motor cortex of monkeys, to learn how motor cortical activity relates to movement and to create new algorithms to decode this activity. ... Read more

Part-based Body Models

Human pose and shape estimation can be seen as a proxy for a wide range of problems in object representation and recognition. Humans are complex and articulated, appear in images in a variety of clothing, and come in a wide range... Read more

Intrinsic Depth

We formulate the estimation of dense depth maps from video sequences as a problem of intrinsic image estimation. Our approach synergistically integrates the estimation of multiple intrinsic images including depth, albedo, shading,... Read more

Lie Bodies

Three-dimensional object shape is commonly represented in terms of deformations of a triangular mesh from an exemplar shape. In particular, statistical generative models of human shape deformation are widely used in computer... Read more

Layers, Time and Segmentation

The segmentation of scenes into regions of coherent structure and the estimation of image motion are fundamental problems in computer vision which are often treated separately. When available, motion provides an important cue for... Read more

Understanding Action Recognition (JHMDB)

Although action recognition in videos is widely studied, current methods often fail on real-world datasets. Many recent approaches improve accuracy and robustness to cope with challenging video sequences, but it is often unclear what... Read more

Intrinsic Video

Intrinsic images such as albedo and shading are valuable for later stages of visual processing. Previous methods for extracting albedo and shading use either single images or images together with depth data. Instead, we defineintrinsic... Read more

Intrinsic Images

The problem of decomposing an image into its different intrinsic layers, such as shading, reflectance, and shape components, is one of the fundamental problems of computer vision. We believe that many computer vision tasks will benefit... Read more

Action Recognition with Tracking

Vision-based human motion analysis attempts to understand the movements of the human body using computer vision and machine learning techniques. The movements of the body can be interpreted on a physical level through pose estimation,... Read more

Neural Control of Grasping

In this project we ask how objects are represented for grasping, how this representation evolves over time and how it is affected by context. This work addresses questions in basic neuroscience but has implications for machine... Read more

Flowing Puppets

We approach the problem of estimating the pose of charactres in TV shows video sequences. Our approach is based on optical flow. People are moving entities, and their motion has distinctive characteristics. Often the whole body has a... Read more

Faces

Due to its relevance for many applications like human computer interaction or face analysis, head pose estimation or facial feature point detection are very active areas in computer vision. Recent state-of-the-art methods have reported... Read more

Deformable Structures

We address the problem of estimating the pose of people in images and video using a new deformable 2D model of the human body. The prevailing approach to this problem uses Pictorial Structures (PS) models, which define a probabilistic... Read more

Model-based Anthropometry

Extracting anthropometric or tailoring measurements from 3D human body scans is important for applications such as virtual try-on, custon clothing, and online sizing. Existing commercial solutions identify anatomical landmarks on... Read more

Modeling 3D Human Breathing

Modeling how the human body deforms during breathing is important for the realistic animation of lifelike 3D avatars. We learn a model of body shape deformations due to breathing for different breathing types and provide simple animation... Read more

Optical flow in the LGN

As we move through our visual environment, the spatial and temporal pattern of light that enters our eyes is strongly influenced by the properties of objects within the environment, their motion relative to each other, and our... Read more

FlowCap

We estimate 2D human pose from video using only optical flow. The key insight is that dense optical flow can provide information about 2D body pose. Like range data, flow is largely invariant to appearance but unlike depth it can be... Read more

Smooth Loops from Unconstrained Video

Converting unconstrained video sequences into videos that loop seamlessly is an extremely challenging problem. In this work, we take the first steps towards automating this process by focusing on an important subclass of videos... Read more

PCA Flow

Despite decades of research, a fast optical flow algorithm, that is also accurate, remains an elusive goal. While recent optical flow methods such as DeepFlow are highly accurate, for many applications speed is often as important. Most...

Read more

Efficient and Scalable Inference

Machine Learning is an important tool for Computer Vision. After the success of Deep Neural Networks(DNN)s in image classification tasks, many other tasks were solved using DNNs. However, working with deep learning, researchers are... Read more

Motion Blur in Layers

Videos contain complex spatially-varying motion blur due to the combination of object motion, camera motion, and depth variation with finite shutter speeds. Existing methods to estimate optical flow, deblur the images, and segment the... Read more

Facade Segmentation

Paper-1: Efficient Facade Segmentation using Auto Context Authors: Varun Jampani*, Raghudeep Gadde* and Peter V. Gehler (*equal contribution) Abstract: In this paper we propose a system for the problem of facade segmentation. Building... Read more

Smooth Metric Learning

Statistics relies on measuring distances. This simple observation has lead to the idea that the distance measure should change locally to better adapt to the problem. With this strategy even simple classifiers can be competitive with the... Read more

Robust PCA

There are several ways to approach robust principal component analysis (RPCA). In early work [

] we noted that PCA constructs a linear subspace that minimizes least squares reconstruction error of the data. Since least... Read more

3D Recognition

The ability to recognize and categorize objects in any type of visual scene is an integral part of scene recognition systems. While for constrained scenarios, like face detection, this problem has largely been solved, the general case of... Read more

Object Detection

Object detection for real world applications is still a challenging problem. While increased data can partly solve the problem, the ability of detectors to process large data sets in reasonable time becomes another important issue... Read more