Traditional
PatchMatch A Randomized Correspondence Algorithm for Structural Image Editing - SIGGRAPH 2009
This paper presents interactive image editing tools using a new randomized algorithm for quickly finding approximate nearest-neighbor matches between image patches. Previous research in graphics and vision has leveraged such nearest-neighbor searches to provide a variety of high-level digital image editing tools. However, the cost of computing a field of such matches for an entire image has eluded previous efforts to provide interactive performance. Our algorithm offers substantial performance improvements over the previous state of the art (20-100x), enabling its use in interactive editing tools. The key insights driving the algorithm are that some good patch matches can be found via random sampling, and that natural coherence in the imagery allows us to propagate such matches quickly to surrounding areas. We offer theoretical analysis of the convergence properties of the algorithm, as well as empirical and practical evidence for its high quality and performance. This one simple algorithm forms the basis for a variety of tools – image retargeting, completion and reshuffling – that can be used together in the context of a high-level image editing application. Finally, we propose additional intuitive constraints on the synthesis process that offer the user a level of control unavailable in previous methods.
Learning-based
Globally and locally consistent image completion - Iizuka - SIGGRAPH 2017
Overview of network architecture It consists of a completion network and two auxiliary context discriminator networks that are used only for training the completion network and are not used during the testing. The global discriminator network takes the entire image as input, while the local discriminator network takes only a small region around the completed area as input. Both discriminator networks are trained to determine if an image is real or completed by the completion network, while the completion network is trained to fool both discriminator networks.
Generative Image Inpainting with Contextual Attention - Yu - CVPR 2018
Overview of our improved generative inpainting framework. The coarse network is trained with reconstruction loss explicitly, while the refinement network is trained with reconstruction loss, global and local WGAN-GP adversarial loss.
Illustration of the contextual attention layer. Firstly we use convolution to compute matching score of foreground patches with background patches (as convolu- tional filters). Then we apply softmax to compare and get attention score for each pixel. Finally we reconstruct fore- ground patches with background patches by performing de- convolution on attention score. The contextual attention layer is differentiable and fully-convolutional.
Based on coarse result from the first encoder- decoder network, two parallel encoders are introduced and then merged to single decoder to get inpainting result. For visualization of attention map, color indicates relative loca- tion of the most interested background patch for each pixel in foreground. For examples, white (center of color coding map) means the pixel attends on itself, pink on bottom-left, green means on top-right.
Pluralistic Image Completion - CVPR 2019
Overview of our architecture with two parallel pipelines. The reconstructive pipeline (yellow line) combines information from Im and Ic, which is used only for training. The generative pipeline (blue line) infers the conditional distribution of hidden regions, that can be sampled during testing. Both representation and generation networks share identical weights.
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning - Nazeri - ICCV 2019
The spirit: “lines first, color next”, which is partly in- spired by our understanding of how artists work.
Appendix
A reference paper list for image inpainting(from geekyutao/Image-Inpainting)
Traditional Methods
| Year | Proceeding | Title | Comment |
|---|---|---|---|
| 2000 | SIGGRAPH 2000 | Image Inpainting [pdf] | Diffusion-based |
| 2001 | TIP 2001 | Filling-in by joint interpolation of vector fields and gray levels [pdf] | Diffusion-based |
| 2001 | CVPR 2001 | Navier-stokes, fluid dynamics, and image and video inpainting [pdf] | |
| 2001 | SIGGRAPH 2001 | Image Quilting for Texture Synthesis and Transfer [pdf] | |
| 2001 | SIGGRAPH 2001 | Synthesizing Natural Textures [pdf] | |
| 2002 | EJAM 2002 | Digital inpainting based on the mumford–shah–euler image model [pdf] | Diffusion-based |
| 2003 | CVPR 2003 | Object removal by exemplar-based inpainting [pdf] | |
| 2003 | TIP 2003 | Simultaneous structure and texture image inpainting [pdf] | Diffusion-based |
| 2003 | TIP 2003 | Structure and Texture Filling-In of Missing Image Blocks in Wireless Transmission and Compression Applications [pdf] | |
| 2003 | ICCV 2003 | Learning How to Inpaint from Global Image Statistics [pdf] | Diffusion-based |
| 2003 | TOG 2003 | Fragment-based image completion [pdf] | Patch-based |
| 2004 | TIP 2004 | Region Filling and Object Removal by Exemplar-Based Image Inpainting [pdf] | Patch-based; Inpainting order |
| 2004 | TPAMI 2004 | Space-Time Video Completion [pdf] | |
| 2005 | SIGGRAPH 2005 | Image Completion with Structure Propagation [pdf] | Patch-based |
| 2006 | ISCS 2006 | Image Compression with Structure Aware Inpainting [pdf] | |
| 2007 | TOG 2007 | Scene completion using millions of photographs [pdf] | |
| 2007 | CSVT 2007 | Image Compression With Edge-Based Inpainting [pdf] | Diffusion-based |
| 2008 | CVPR 2008 | Summarizing Visual Data Using Bidirectional Similarity [pdf] | |
| 2009 | SIGGRAPH 2009 | PatchMatch: a randomized correspondence algorithm for structural image editing [pdf] | Patch-based |
| 2010 | TIP 2010 | Image inpainting by patch propagation using patch sparsity [pdf] | Patch-based |
| 2011 | FTCGV 2011 | Structured learning and prediction in computer vision [pdf] | |
| 2011 | ICIP 2011 | Examplar-based inpainting based on local geometry [pdf] | Inpainting order |
| 2012 | TOG 2012 | Combining inconsistent images using patch-based synthesis[pdf] | Patch-based |
| 2014 | TOG 2014 | Image completion using Planar structure guidance [pdf] | Patch-based |
| 2014 | TVCG 2014 | High-Quality Real-Time Video Inpainting with PixMix [pdf] | Video |
| 2014 | SIAM 2014 | Video inpainting of complex scenes [pdf] | Video |
| 2015 | TIP 2015 | Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting [pdf] | |
| 2015 | TIP 2015 | Exemplar-Based Inpainting: Technical Review and New Heuristics for Better Geometric Reconstructions [pdf] | |
| 2016 | TOG 2016 | Temporally coherent completion of dynamic video [pdf] | Video |
Deep-Learning-based Method
| Year | Proceeding | Title | Comment |
|---|---|---|---|
| 2012 | NIPS 2012 | Image denoising and inpainting with deep neural networks [pdf] | |
| 2014 | GCPR 2014 | Mask-specific inpainting with deep neural networks [pdf] | |
| 2014 | NIPS 2014 | Deep Convolutional Neural Network for Image Deconvolution [pdf] | |
| 2015 | NIPS 2015 | Shepard Convolutional Neural Networks [pdf] [code] | |
| 2016 | CVPR 2016 | Context Encoders: Feature Learning by Inpainting [pdf] [code] | |
| 2016 | SIGGRAPH 2016 | High-resolution multi-scale neural texture synthesis [pdf] | |
| 2017 | CVPR 2017 | Semantic image inpainting with deep generative models [pdf] [code] | |
| 2017 | CVPR 2017 | High-resolution image inpainting using multi-scale neural patch synthesis [pdf] [code] | |
| 2017 | CVPR 2017 | Generative Face Completion [pdf] [code] | |
| 2017 | SIGGRAPH 2017 | Globally and Locally Consistent Image Completion [pdf] [code] | |
| 2018 | CVPR 2018 | Generative Image Inpainting with Contextual Attention [pdf] [code] | |
| 2018 | CVPR 2018 | Natural and Effective Obfuscation by Head Inpainting [pdf] | |
| 2018 | CVPR 2018 | Eye In-Painting With Exemplar Generative Adversarial Networks [pdf] [code] | |
| 2018 | CVPR 2018 | UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition [pdf] | |
| 2018 | CVPR 2018 | Disentangling Structure and Aesthetics for Style-aware Image Completion [pdf] | |
| 2018 | ECCV 2018 | Image Inpainting for Irregular Holes Using Partial Convolutions [pdf] [code] | |
| 2018 | ECCV 2018 | Contextual-based Image Inpainting: Infer, Match, and Translate [pdf] | |
| 2018 | ECCV 2018 | Shift-Net: Image Inpainting via Deep Feature Rearrangement [pdf] [code] | |
| 2018 | NIPS 2018 | Image Inpainting via Generative Multi-column Convolutional Neural Networks [pdf] [code] | |
| 2018 | TOG 2018 | Faceshop: Deep sketch-based face image editing [pdf] | |
| 2018 | ACM MM 2018 | Structural inpainting [pdf] | |
| 2018 | ACM MM 2018 | Semantic Image Inpainting with Progressive Generative Networks [pdf] [code] | |
| 2018 | BMVC 2018 | SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting [pdf] | |
| 2018 | BMVC 2018 | MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesi [pdf] [code] | |
| 2018 | ACCV 2018 | Face Completion iwht Semantic Knowledge and Collaborative Adversarial Learning [pdf] | |
| 2018 | ICASSP 2018 | Edge-Aware Context Encoder for Image Inpainting [pdf] | |
| 2018 | ICPR 2018 | Deep Structured Energy-Based Image Inpainting [pdf] [code] | |
| 2018 | AISTATS 2019 | Probabilistic Semantic Inpainting with Pixel Constrained CNNs [pdf] | |
| 2019 | ICRA 2019 | Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space [pdf] | |
| 2019 | AAAI 2019 | Video Inpainting by Jointly Learning Temporal Structure and Spatial Details [pdf] | Video |
| 2019 | CVPR 2019 | Pluralistic Image Completion [pdf] [code] [project] | |
| 2019 | CVPR 2019 | Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image [pdf] | |
| 2019 | CVPR 2019 | Foreground-aware Image Inpainting [pdf] | |
| 2019 | CVPR 2019 | Privacy Protection in Street-View Panoramas using Depth and Multi-View Imagery [pdf] | |
| 2019 | CVPR 2019 | Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting [pdf] [code] | |
| 2019 | CVPR 2019 | Deep Flow-Guided Video Inpainting [pdf] [project] | Video |
| 2019 | CVPR 2019 | Deep video inapinting [pdf] | Video |
| 2019 | CVPR Workshop 2019 | VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal [pdf] | Video |
| 2019 | TNNLS 2019 | PEPSI++: Fast and Lightweight Network for Image Inpainting [pdf] | |
| 2019 | ACM MM 2019 | Progressive Image Inpainting with Full-Resolution Residual Network [pdf] | |
| 2019 | ICCV 2019 | EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning [pdf] [code] | |
| 2019 | ICCV 2019 | Coherent Semantic Attention for Image Inpainting [pdf] [code] | |
| 2019 | ICCV 2019 | StructureFlow: Image Inpainting via Structure-aware Appearance Flow [pdf] [code] | |
| 2019 | ICCV 2019 | Image Inpainting with Learnable Bidirectional Attention Maps [pdf] [code] | |
| 2018 | arXiv:1801.07632 | High Resolution Face Completion with Multiple Controllable Attributes via Fully End-to-End Progressive Generative Adversarial Networks [pdf] | |
| 2018 | arXiv:1803.07422 | Patch-Based Image Inpainting with Generative Adversarial Networks [pdf] [code] | |
| 2018 | arXiv:1806.03589 | Free-Form Image Inpainting with Gated Convolution [pdf] [project] | Gated Conv |
| 2018 | arXiv:1808.04432 | X-GANs: Image Reconstruction Made Easy for Extreme Cases [pdf] | |
| 2018 | arXiv:1811.03721 | Learning Energy Based Inpainting for Optical Flow [pdf] | |
| 2018 | arXiv:1811.07104 | On Hallucinating Context and Background Pixels from a Face Mask using Multi-scale GANs [pdf] | |
| 2018 | arXiv:1811.09012 | Multi-View Inpainting for RGB-D Sequence [pdf] | |
| 2018 | arXiv:1812.01458 | Deep Inception Generative network for Cognitive Image Inpainting [pdf] | |
| 2019 | arXiv:1901.03396 | Detecting Overfitting of Deep Generative Networks via Latent Recovery [pdf] | |
| 2019 | arXiv:1902.01096 | Compatible and Diverse Fashion Image Inpainting [pdf] | |
| 2019 | arXiv:1902.06838 | SC-FEGAN: Face Editing Generative Adversarial Network with User’s Sketch and Color [pdf] [code] | |
| 2019 | arXiv:1902.09225 | Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation [pdf] | |
| 2019 | arXiv:1903.00450 | Multi-Object Representation Learning with Iterative Variational Inference [pdf] | |
| 2019 | arXiv:1903.04842 | Unsupervised motion saliency map estimation based on optical flow inpainting [pdf] | |
| 2019 | arXiv:1903.10885 | Learning Quadrangulated Patches For 3D Shape Processing [pdf] | |
| 2019 | arXiv:1904.10247 | Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN [pdf] | Video |
| 2019 | arXiv:1904.10795 | Graph-based Inpainting for 3D Dynamic Point Clouds [pdf] | Point Cloud |
| 2019 | arXiv:1905.02882 | Frame-Recurrent Video Inpainting by Robust Optical Flow Inference [pdf] | Video |
| 2019 | arXiv:1905.02949 | Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence [pdf] | Video |
| 2019 | arXiv:1905.13066 | Align-and-Attend Network for Globally and Locally Coherent Video Inpainting [pdf] | Video |
| 2019 | arXiv:1906.00884 | Fashion Editing with Multi-scale Attention Normalization [pdf] |