EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning - Nazeri - 2019 - PyTorch

Info

Title: EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Task: Image Inpainting
Author: K. Nazeri, E. Ng, T. Joseph, F. Qureshi, and M. Ebrahimi
Date: Jan. 2019
Arxiv: 1901.00212
Published: ICCV 2019 Workshop

Highlights & Drawbacks

Interactive sketch editing for image completion
A two-stage adversarial model that comprises of an edge generator followed by an image completion network.

Motivation & Design

The spirit: “lines first, color next”, which is partly in- spired by our understanding of how artists work.

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

Edge Generator

Feature-Matching loss: $\mathcal{L}_{F M}=\mathbb{E}\left[\sum_{i=1}^{L} \frac{1}{N_{i}}\left\|D_{1}^{(i)}\left(\mathbf{C}_{g t}\right)-D_{1}^{(i)}\left(\mathbf{C}_{p r e d}\right)\right\|_{1}\right]$

Adversarial loss: $\begin{aligned} \mathcal{L}_{a d v, 1}=\mathbb{E}_{\left(\mathbf{C}_{g t}, \mathbf{I}_{g r a y}\right)}\left[\log D_{1}\left(\mathbf{C}_{g t}, \mathbf{I}_{g r a y}\right)\right] & ] \\ &+\mathbb{E}_{\mathbf{I}_{g r a y}} \log \left[1-D_{1}\left(\mathbf{C}_{p r e d}, \mathbf{I}_{g r a y}\right)\right] \end{aligned}$

Completion Network

Adversarial loss: $\begin{aligned} \mathcal{L}_{a d v, 2}=& \mathbb{E}_{\left(\mathbf{I}_{g t}, \mathbf{C}_{c o m p}\right)}\left[\log D_{2}\left(\mathbf{I}_{g t}, \mathbf{C}_{c o m p}\right)\right] \\ &+\mathbb{E}_{\mathbf{C}_{c o m p}} \log \left[1-D_{2}\left(\mathbf{I}_{p r e d}, \mathbf{C}_{c o m p}\right)\right] \end{aligned}$

Perceptual loss: $\mathcal{L}_{p e r c}=\mathbb{E}\left[\sum_{i} \frac{1}{N_{i}}\left\|\phi_{i}\left(\mathbf{I}_{g t}\right)-\phi_{i}\left(\mathbf{I}_{p r e d}\right)\right\|_{1}\right]$

Style loss: $\mathcal{L}_{s t y l e}=\mathbb{E}_{j}\left[\left\|G_{j}^{\phi}\left(\tilde{\mathbf{I}}_{p r e d}\right)-G_{j}^{\phi}\left(\tilde{\mathbf{I}}_{g t}\right)\right\|_{1}\right]$

Performance & Ablation Study

Quality Results

Left to Right: Original image, input image, generated edges, inpainted results without any post-processing. EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

Quantitative results over Places2 with models

Left to right: Contextual Attention (CA), Globally and Locally Consistent Image Completion (GLCIC), Partial Convolu- tion (PConv) , G1 and G2 (Ours), G2 only with Canny edges (Canny). The best result of each row is boldfaced except for Canny. EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning