Infusion: Internal Diffusion for Inpainting of Dynamic Textures and Complex Motion

Simple item page
dc.contributor.authorCherel, Nicolasen_US
dc.contributor.authorAlmansa, Andrésen_US
dc.contributor.authorGousseau, Yannen_US
dc.contributor.authorNewson, Alasdairen_US
dc.contributor.editorBousseau, Adrienen_US
dc.contributor.editorDay, Angelaen_US
dc.date.accessioned2025-05-09T09:15:51Z
dc.date.available2025-05-09T09:15:51Z
dc.date.issued2025
dc.description.abstractVideo inpainting is the task of filling a region in a video in a visually convincing manner. It is very challenging due to the high dimensionality of the data and the temporal consistency required for obtaining convincing results. Recently, diffusion models have shown impressive results in modeling complex data distributions, including images and videos. Such models remain nonetheless very expensive to train and to perform inference with, which strongly reduce their applicability to videos, and yields unreasonable computational loads. We show that in the case of video inpainting, thanks to the highly auto-similar nature of videos, the training data of a diffusion model can be restricted to the input video and still produce very satisfying results. With this internal learning approach, where the training data is limited to a single video, our lightweight models perform very well with only half a million parameters, in contrast to the very large networks with billions of parameters typically found in the literature. We also introduce a new method for efficient training and inference of diffusion models in the context of internal learning, by splitting the diffusion process into different learning intervals corresponding to different noise levels of the diffusion process. We show qualitative and quantitative results, demonstrating that our method reaches or exceeds state of the art performance in the case of dynamic textures and complex dynamic backgrounds.en_US
dc.description.number2
dc.description.sectionheadersFix it in Post: Image and Video Synthesis and Analysis
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume44
dc.identifier.doi10.1111/cgf.70070
dc.identifier.issn1467-8659
dc.identifier.pages14 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.70070
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf70070
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.rightsAttribution 4.0 International License
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectCCS Concepts: Computing methodologies → Image processing
dc.subjectComputing methodologies → Image processing
dc.titleInfusion: Internal Diffusion for Inpainting of Dynamic Textures and Complex Motionen_US
Files
Original bundle
Now showing 1 - 5 of 17
Thumbnail Image
Name:
cgf70070.pdf
Size:
10.45 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
ablation_study.mp4
Size:
2.9 MB
Format:
Video MP4
Download
No Thumbnail Available
Name:
average.mp4
Size:
6.59 MB
Format:
Video MP4
Download
No Thumbnail Available
Name:
c0215.mp4
Size:
2.05 MB
Format:
Video MP4
Download
No Thumbnail Available
Name:
c0222.mp4
Size:
7.78 MB
Format:
Video MP4
Download
Collections
EG 2025 - Full Papers - CGF 44-Issue 2
44-Issue 2