[go: up one dir, main page]

GB2600869B - Video prediction using one or more neural networks - Google Patents

Video prediction using one or more neural networks Download PDF

Info

Publication number
GB2600869B
GB2600869B GB2201888.1A GB202201888A GB2600869B GB 2600869 B GB2600869 B GB 2600869B GB 202201888 A GB202201888 A GB 202201888A GB 2600869 B GB2600869 B GB 2600869B
Authority
GB
United Kingdom
Prior art keywords
neural networks
video prediction
prediction
video
neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
GB2201888.1A
Other versions
GB2600869A (en
GB202201888D0 (en
Inventor
Shih Kevin
Dundar Aysegul
Garg Animesh
Pottorff Robert
Tao Andrew
Catanzaro Bryan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nvidia Corp
Original Assignee
Nvidia Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nvidia Corp filed Critical Nvidia Corp
Publication of GB202201888D0 publication Critical patent/GB202201888D0/en
Publication of GB2600869A publication Critical patent/GB2600869A/en
Application granted granted Critical
Publication of GB2600869B publication Critical patent/GB2600869B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
GB2201888.1A 2019-09-03 2020-08-19 Video prediction using one or more neural networks Active GB2600869B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/558,620 US11902705B2 (en) 2019-09-03 2019-09-03 Video prediction using one or more neural networks
PCT/US2020/046982 WO2021045905A1 (en) 2019-09-03 2020-08-19 Video prediction using one or more neural networks

Publications (3)

Publication Number Publication Date
GB202201888D0 GB202201888D0 (en) 2022-03-30
GB2600869A GB2600869A (en) 2022-05-11
GB2600869B true GB2600869B (en) 2024-02-21

Family

ID=72291146

Family Applications (1)

Application Number Title Priority Date Filing Date
GB2201888.1A Active GB2600869B (en) 2019-09-03 2020-08-19 Video prediction using one or more neural networks

Country Status (5)

Country Link
US (1) US11902705B2 (en)
CN (1) CN114303156A (en)
DE (1) DE112020004167T5 (en)
GB (1) GB2600869B (en)
WO (1) WO2021045905A1 (en)

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019097784A1 (en) * 2017-11-16 2019-05-23 ソニー株式会社 Information processing device, information processing method, and program
US11995854B2 (en) * 2018-12-19 2024-05-28 Nvidia Corporation Mesh reconstruction using data-driven priors
JP7285203B2 (en) * 2019-11-28 2023-06-01 株式会社日立製作所 Generation device, data analysis system, generation method, and generation program
CN111241398B (en) * 2020-01-10 2023-07-25 百度在线网络技术（北京）有限公司 Data prefetching method, device, electronic equipment and computer readable storage medium
WO2021176985A1 (en) * 2020-03-05 2021-09-10 ソニーセミコンダクタソリューションズ株式会社 Signal processing device, signal processing method, and program
CN111787323B (en) * 2020-05-23 2021-09-03 清华大学 Variable bit rate generation type compression method based on counterstudy
US11380121B2 (en) * 2020-08-25 2022-07-05 Sony Group Corporation Full skeletal 3D pose recovery from monocular camera
CN112584076B (en) * 2020-12-11 2022-12-06 北京百度网讯科技有限公司 Video frame interpolation method and device and electronic equipment
US12223719B2 (en) * 2020-12-11 2025-02-11 Korea University Research And Business Foundation Apparatus and method for prediction of video frame based on deep learning
US11922640B2 (en) * 2021-03-08 2024-03-05 Toyota Research Institute, Inc. Semi-supervised 3D object tracking in videos via 2D semantic keypoints
US20220392099A1 (en) * 2021-05-28 2022-12-08 Disney Enterprises, Inc. Stable pose estimation with analysis by synthesis
US12494056B2 (en) 2021-09-30 2025-12-09 Lemon Inc. Social networking based on asset items
US11763496B2 (en) * 2021-09-30 2023-09-19 Lemon Inc. Social networking based on asset items
CN113822228B (en) * 2021-10-27 2024-03-22 南京大学 User expression recognition method and system based on continuous learning
US12250150B2 (en) * 2021-12-28 2025-03-11 International Business Machines Corporation AI-based compensation of resource constrained communication
US12097431B2 (en) * 2022-02-11 2024-09-24 Electronic Arts Inc. Goal driven animation
US20230281877A1 (en) * 2022-03-04 2023-09-07 Eduardo CORRAL-SOTO Systems and methods for 3d point cloud densification
CN116721437A (en) * 2022-03-07 2023-09-08 竞舞娱乐私人有限公司 Devices and methods for estimating body shape and posture
US20240095447A1 (en) * 2022-06-22 2024-03-21 Nvidia Corporation Neural network-based language restriction
US12488481B2 (en) * 2022-08-12 2025-12-02 Meta Platforms, Inc. Video reconstruction from videos with ultra-low frame-per-second
US11727618B1 (en) * 2022-08-25 2023-08-15 xNeurals Inc. Artificial intelligence-based system and method for generating animated videos from an audio segment
WO2024064370A2 (en) * 2022-09-23 2024-03-28 Apple Inc. Deep learning based causal image reprojection for temporal supersampling in ar/vr systems
US12318661B2 (en) 2022-10-18 2025-06-03 Tonal Systems, Inc. Exercise guidance using multi-modal data
US20240169698A1 (en) * 2022-11-23 2024-05-23 Logitech Europe S.A. Object detection using artificial intelligence
US20240193812A1 (en) * 2022-12-07 2024-06-13 Htc Corporation Hand pose construction method, electronic device, and non-transitory computer readable storage medium
US20240289975A1 (en) * 2023-02-24 2024-08-29 Qualcomm Incorporated Pose prediction of objects for extended reality systems
CN116863379B (en) * 2023-07-11 2025-03-07 杭州电子科技大学 Video prediction defense method based on space-time self-attention single-step disturbance
US20250071354A1 (en) * 2023-08-21 2025-02-27 Google Llc Systems and methods for generating transitions between videos
US20250190866A1 (en) * 2023-12-12 2025-06-12 AtomBeam Technologies Inc. Multimodal data processing and generation system using vq-vae and latent transformer

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100792016B1 (en) * 2006-07-25 2008-01-04 한국항공대학교산학협력단 Character-based video summary device using audio and video information and method thereof
EP3335195A2 (en) * 2015-08-14 2018-06-20 Metail Limited Methods of generating personalized 3d head models or 3d body models
US10799150B2 (en) * 2015-12-22 2020-10-13 Picterus As Image based bilirubin determination
JP6635848B2 (en) * 2016-03-31 2020-01-29 ソフトバンク株式会社 Three-dimensional video data generation device, three-dimensional video data generation program, and method therefor
US10096125B1 (en) * 2017-04-07 2018-10-09 Adobe Systems Incorporated Forecasting multiple poses based on a graphical image
US10460175B1 (en) * 2017-06-15 2019-10-29 Amazon Technologies, Inc. Deep learning processing of video
US10373332B2 (en) * 2017-12-08 2019-08-06 Nvidia Corporation Systems and methods for dynamic facial analysis using a recurrent neural network
CN108447094B (en) * 2018-03-20 2020-07-28 清华大学 Method and system for estimating attitude of monocular color camera
CN108537820B (en) * 2018-04-18 2021-02-09 图灵人工智能研究院（南京）有限公司 Dynamic prediction method, system and applicable equipment
CN109033946A (en) * 2018-06-08 2018-12-18 东南大学 Merge the estimation method of human posture of directional diagram
US10861170B1 (en) * 2018-11-30 2020-12-08 Snap Inc. Efficient human pose tracking in videos
CN109754015B (en) * 2019-01-02 2021-01-26 京东方科技集团股份有限公司 Neural network and related methods, media and devices for multi-label recognition of paintings

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
DOMINIK LORENZ ET AL, "Unsupervised Part-Based Disentangling of Object Shape and Appearance", 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 1 June 2019 (2019-06-01), pages 10947-10956, doi:10.1109/CVPR.2019.01121, ISBN 978-1-7281-3293-8 abstract page 1 - page 8 figures *
Kevin J Shih ET AL, "Video Interpolation and Prediction with Unsupervised Landmarks", 6 September 2019 (20190906), retrieved from the internet:URL: https://arxiv.org/pdf/1909.02749.pdf, [retrieved on 2020-11-04], abstract page 1 - page 9 figures 1-8 *
SAMBREKAR PRATHMESH ET AL, "Movie Frame Prediction Using Convolutional Long Short Term Memory", 2019 2ND INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTING, INSTRUMENTATION AND CONTROL TECHNOLOGIES (ICICICT), IEEE, vol. 1, 5 July 2019 (2019-07-05), pages 1-5 DOI: 10.1109/ICICICT46008.2019.8993289 ISB *
TOMAS JAKAB ET AL, "Unsupervised Learning of Object Landmarks through Conditional Image Generation", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 20 June 2018(2018-06-20), abstract page 1 - page 9 figures 1-14 *

Also Published As

Publication number Publication date
WO2021045905A1 (en) 2021-03-11
US11902705B2 (en) 2024-02-13
DE112020004167T5 (en) 2022-07-14
CN114303156A (en) 2022-04-08
GB2600869A (en) 2022-05-11
US20210064925A1 (en) 2021-03-04
GB202201888D0 (en) 2022-03-30

Similar Documents

Publication Publication Date Title
GB2600869B (en) Video prediction using one or more neural networks
GB2623893B (en) Video upsampling using one or more neural networks
GB2603084B (en) Pose determination using one or more neural networks
GB202201887D0 (en) Image generation using one or more neural networks
EP4000015A4 (en) Occupancy prediction neural networks
GB2600866B (en) Player analysis using one or more neural networks
GB2600357B (en) Content recommendations using one or more neural networks
GB2600793B (en) Video synthesis using one or more neural networks
SG11202103016RA (en) Bi-prediction for video coding
GB201810897D0 (en) Phenotype prediction
GB202112524D0 (en) Image enhancement using one or more neural networks
IL292873A (en) Spatio-temporal-interactive networks
EP3777158A4 (en) Resolution-adaptive video coding
GB201905400D0 (en) Video coding technology
GB201902449D0 (en) Range prediction
GB2601213B (en) Interaction determination using one or more neural networks
GB201805973D0 (en) Neural network
HK40061411A (en) Residual semi-recurrent neural networks
GB201914377D0 (en) Shoutout video
GB201918099D0 (en) Video coding technology
GB201916090D0 (en) Video coding technology
GB201915553D0 (en) Video Coding technology
GB201914634D0 (en) Video coding technology
GB201914414D0 (en) Video coding technology
GB201914215D0 (en) Video coding technology