Implementation of Video Diffusion Models, Jonathan Ho's new paper extending DDPMs to Video Generation - in Pytorch. Implementation of Video Diffusion Models, Jonathan Ho's new paper extending DDPMs to Video Generation - in Pytorch. It uses a special space-time factored U-net, extending generation from 2D images to 3D videos. 14k for difficult moving mnist (converging much faster and better than NUWA) - wip. Any new developments for text-to-video synthesis will be centralized at Imagen-pytorch. For conditioning on text, they derived text embeddings by first passing the tokenized text through BERT-large. You can also directly pass in the descriptions of the video as strings, if you plan on using BERT-base for text conditioning. This repository also contains a handy Trainer class for training on a folder of gifs. Each gif must be of the correct dimensions image_size and num_frames.
Features
- From 2D images to 3D videos
- Co-training Images and Video
- Sample videos (as gif files) will be saved to ./results periodically, as are the diffusion model parameters
- You can also directly pass in the descriptions of the video as strings
- Implementation of Video Diffusion Models, Jonathan Ho's new paper
- It uses a special space-time factored U-net
Project Samples
