Image Inpainting: From PatchMatch to Pluralistic



PatchMatch A Randomized Correspondence Algorithm for Structural Image Editing - SIGGRAPH 2009

CleanShot 2019-09-22 at 14.59.37@2x

This paper presents interactive image editing tools using a new randomized algorithm for quickly finding approximate nearest-neighbor matches between image patches. Previous research in graphics and vision has leveraged such nearest-neighbor searches to provide a variety of high-level digital image editing tools. However, the cost of computing a field of such matches for an entire image has eluded previous efforts to provide interactive performance. Our algorithm offers substantial performance improvements over the previous state of the art (20-100x), enabling its use in interactive editing tools. The key insights driving the algorithm are that some good patch matches can be found via random sampling, and that natural coherence in the imagery allows us to propagate such matches quickly to surrounding areas. We offer theoretical analysis of the convergence properties of the algorithm, as well as empirical and practical evidence for its high quality and performance. This one simple algorithm forms the basis for a variety of tools – image retargeting, completion and reshuffling – that can be used together in the context of a high-level image editing application. Finally, we propose additional intuitive constraints on the synthesis process that offer the user a level of control unavailable in previous methods.


Globally and locally consistent image completion - Iizuka - SIGGRAPH 2017

Globally and locally consistent image completion - Iizuka - SIGGRAPH 2017

Overview of network architecture It consists of a completion network and two auxiliary context discriminator networks that are used only for training the completion network and are not used during the testing. The global discriminator network takes the entire image as input, while the local discriminator network takes only a small region around the completed area as input. Both discriminator networks are trained to determine if an image is real or completed by the completion network, while the completion network is trained to fool both discriminator networks.

Globally and locally consistent image completion - Iizuka - SIGGRAPH 2017

Generative Image Inpainting with Contextual Attention - Yu - CVPR 2018

Overview of our improved generative inpainting framework. The coarse network is trained with reconstruction loss explicitly, while the refinement network is trained with reconstruction loss, global and local WGAN-GP adversarial loss.

Generative Image Inpainting with Contextual Attention

Illustration of the contextual attention layer. Firstly we use convolution to compute matching score of foreground patches with background patches (as convolu- tional filters). Then we apply softmax to compare and get attention score for each pixel. Finally we reconstruct fore- ground patches with background patches by performing de- convolution on attention score. The contextual attention layer is differentiable and fully-convolutional.

Generative Image Inpainting with Contextual Attention

Based on coarse result from the first encoder- decoder network, two parallel encoders are introduced and then merged to single decoder to get inpainting result. For visualization of attention map, color indicates relative loca- tion of the most interested background patch for each pixel in foreground. For examples, white (center of color coding map) means the pixel attends on itself, pink on bottom-left, green means on top-right.

Generative Image Inpainting with Contextual Attention

Pluralistic Image Completion - CVPR 2019

CleanShot 2019-09-22 at 14.53.16@2x

Overview of our architecture with two parallel pipelines. The reconstructive pipeline (yellow line) combines information from Im and Ic, which is used only for training. The generative pipeline (blue line) infers the conditional distribution of hidden regions, that can be sampled during testing. Both representation and generation networks share identical weights.

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning - Nazeri - ICCV 2019

The spirit: “lines first, color next”, which is partly in- spired by our understanding of how artists work.

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning


A reference paper list for image inpainting(from geekyutao/Image-Inpainting)

Traditional Methods

Year Proceeding Title Comment
2000 SIGGRAPH 2000 Image Inpainting [pdf] Diffusion-based
2001 TIP 2001 Filling-in by joint interpolation of vector fields and gray levels [pdf] Diffusion-based
2001 CVPR 2001 Navier-stokes, fluid dynamics, and image and video inpainting [pdf]  
2001 SIGGRAPH 2001 Image Quilting for Texture Synthesis and Transfer [pdf]  
2001 SIGGRAPH 2001 Synthesizing Natural Textures [pdf]  
2002 EJAM 2002 Digital inpainting based on the mumford–shah–euler image model [pdf] Diffusion-based
2003 CVPR 2003 Object removal by exemplar-based inpainting [pdf]  
2003 TIP 2003 Simultaneous structure and texture image inpainting [pdf] Diffusion-based
2003 TIP 2003 Structure and Texture Filling-In of Missing Image Blocks in Wireless Transmission and Compression Applications [pdf]  
2003 ICCV 2003 Learning How to Inpaint from Global Image Statistics [pdf] Diffusion-based
2003 TOG 2003 Fragment-based image completion [pdf] Patch-based
2004 TIP 2004 Region Filling and Object Removal by Exemplar-Based Image Inpainting [pdf] Patch-based; Inpainting order
2004 TPAMI 2004 Space-Time Video Completion [pdf]  
2005 SIGGRAPH 2005 Image Completion with Structure Propagation [pdf] Patch-based
2006 ISCS 2006 Image Compression with Structure Aware Inpainting [pdf]  
2007 TOG 2007 Scene completion using millions of photographs [pdf]  
2007 CSVT 2007 Image Compression With Edge-Based Inpainting [pdf] Diffusion-based
2008 CVPR 2008 Summarizing Visual Data Using Bidirectional Similarity [pdf]  
2009 SIGGRAPH 2009 PatchMatch: a randomized correspondence algorithm for structural image editing [pdf] Patch-based
2010 TIP 2010 Image inpainting by patch propagation using patch sparsity [pdf] Patch-based
2011 FTCGV 2011 Structured learning and prediction in computer vision [pdf]  
2011 ICIP 2011 Examplar-based inpainting based on local geometry [pdf] Inpainting order
2012 TOG 2012 Combining inconsistent images using patch-based synthesis[pdf] Patch-based
2014 TOG 2014 Image completion using Planar structure guidance [pdf] Patch-based
2014 TVCG 2014 High-Quality Real-Time Video Inpainting with PixMix [pdf] Video
2014 SIAM 2014 Video inpainting of complex scenes [pdf] Video
2015 TIP 2015 Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting [pdf]  
2015 TIP 2015 Exemplar-Based Inpainting: Technical Review and New Heuristics for Better Geometric Reconstructions [pdf]  
2016 TOG 2016 Temporally coherent completion of dynamic video [pdf] Video

Deep-Learning-based Method

Year Proceeding Title Comment
2012 NIPS 2012 Image denoising and inpainting with deep neural networks [pdf]  
2014 GCPR 2014 Mask-specific inpainting with deep neural networks [pdf]  
2014 NIPS 2014 Deep Convolutional Neural Network for Image Deconvolution [pdf]  
2015 NIPS 2015 Shepard Convolutional Neural Networks [pdf] [code]  
2016 CVPR 2016 Context Encoders: Feature Learning by Inpainting [pdf] [code]  
2016 SIGGRAPH 2016 High-resolution multi-scale neural texture synthesis [pdf]  
2017 CVPR 2017 Semantic image inpainting with deep generative models [pdf] [code]  
2017 CVPR 2017 High-resolution image inpainting using multi-scale neural patch synthesis [pdf] [code]  
2017 CVPR 2017 Generative Face Completion [pdf] [code]  
2017 SIGGRAPH 2017 Globally and Locally Consistent Image Completion [pdf] [code]  
2018 CVPR 2018 Generative Image Inpainting with Contextual Attention [pdf] [code]  
2018 CVPR 2018 Natural and Effective Obfuscation by Head Inpainting [pdf]  
2018 CVPR 2018 Eye In-Painting With Exemplar Generative Adversarial Networks [pdf] [code]  
2018 CVPR 2018 UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition [pdf]  
2018 CVPR 2018 Disentangling Structure and Aesthetics for Style-aware Image Completion [pdf]  
2018 ECCV 2018 Image Inpainting for Irregular Holes Using Partial Convolutions [pdf] [code]  
2018 ECCV 2018 Contextual-based Image Inpainting: Infer, Match, and Translate [pdf]  
2018 ECCV 2018 Shift-Net: Image Inpainting via Deep Feature Rearrangement [pdf] [code]  
2018 NIPS 2018 Image Inpainting via Generative Multi-column Convolutional Neural Networks [pdf] [code]  
2018 TOG 2018 Faceshop: Deep sketch-based face image editing [pdf]  
2018 ACM MM 2018 Structural inpainting [pdf]  
2018 ACM MM 2018 Semantic Image Inpainting with Progressive Generative Networks [pdf] [code]  
2018 BMVC 2018 SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting [pdf]  
2018 BMVC 2018 MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesi [pdf] [code]  
2018 ACCV 2018 Face Completion iwht Semantic Knowledge and Collaborative Adversarial Learning [pdf]  
2018 ICASSP 2018 Edge-Aware Context Encoder for Image Inpainting [pdf]  
2018 ICPR 2018 Deep Structured Energy-Based Image Inpainting [pdf] [code]  
2018 AISTATS 2019 Probabilistic Semantic Inpainting with Pixel Constrained CNNs [pdf]  
2019 ICRA 2019 Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space [pdf]  
2019 AAAI 2019 Video Inpainting by Jointly Learning Temporal Structure and Spatial Details [pdf] Video
2019 CVPR 2019 Pluralistic Image Completion [pdf] [code] [project]  
2019 CVPR 2019 Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image [pdf]  
2019 CVPR 2019 Foreground-aware Image Inpainting [pdf]  
2019 CVPR 2019 Privacy Protection in Street-View Panoramas using Depth and Multi-View Imagery [pdf]  
2019 CVPR 2019 Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting [pdf] [code]  
2019 CVPR 2019 Deep Flow-Guided Video Inpainting [pdf] [project] Video
2019 CVPR 2019 Deep video inapinting [pdf] Video
2019 CVPR Workshop 2019 VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal [pdf] Video
2019 TNNLS 2019 PEPSI++: Fast and Lightweight Network for Image Inpainting [pdf]  
2019 ACM MM 2019 Progressive Image Inpainting with Full-Resolution Residual Network [pdf]  
2019 ICCV 2019 EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning [pdf] [code]  
2019 ICCV 2019 Coherent Semantic Attention for Image Inpainting [pdf] [code]  
2019 ICCV 2019 StructureFlow: Image Inpainting via Structure-aware Appearance Flow [pdf] [code]  
2019 ICCV 2019 Image Inpainting with Learnable Bidirectional Attention Maps [pdf] [code]  
2018 arXiv:1801.07632 High Resolution Face Completion with Multiple Controllable Attributes via Fully End-to-End Progressive Generative Adversarial Networks [pdf]  
2018 arXiv:1803.07422 Patch-Based Image Inpainting with Generative Adversarial Networks [pdf] [code]  
2018 arXiv:1806.03589 Free-Form Image Inpainting with Gated Convolution [pdf] [project] Gated Conv
2018 arXiv:1808.04432 X-GANs: Image Reconstruction Made Easy for Extreme Cases [pdf]  
2018 arXiv:1811.03721 Learning Energy Based Inpainting for Optical Flow [pdf]  
2018 arXiv:1811.07104 On Hallucinating Context and Background Pixels from a Face Mask using Multi-scale GANs [pdf]  
2018 arXiv:1811.09012 Multi-View Inpainting for RGB-D Sequence [pdf]  
2018 arXiv:1812.01458 Deep Inception Generative network for Cognitive Image Inpainting [pdf]  
2019 arXiv:1901.03396 Detecting Overfitting of Deep Generative Networks via Latent Recovery [pdf]  
2019 arXiv:1902.01096 Compatible and Diverse Fashion Image Inpainting [pdf]  
2019 arXiv:1902.06838 SC-FEGAN: Face Editing Generative Adversarial Network with User’s Sketch and Color [pdf] [code]  
2019 arXiv:1902.09225 Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation [pdf]  
2019 arXiv:1903.00450 Multi-Object Representation Learning with Iterative Variational Inference [pdf]  
2019 arXiv:1903.04842 Unsupervised motion saliency map estimation based on optical flow inpainting [pdf]  
2019 arXiv:1903.10885 Learning Quadrangulated Patches For 3D Shape Processing [pdf]  
2019 arXiv:1904.10247 Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN [pdf] Video
2019 arXiv:1904.10795 Graph-based Inpainting for 3D Dynamic Point Clouds [pdf] Point Cloud
2019 arXiv:1905.02882 Frame-Recurrent Video Inpainting by Robust Optical Flow Inference [pdf] Video
2019 arXiv:1905.02949 Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence [pdf] Video
2019 arXiv:1905.13066 Align-and-Attend Network for Globally and Locally Coherent Video Inpainting [pdf] Video
2019 arXiv:1906.00884 Fashion Editing with Multi-scale Attention Normalization [pdf]