NTIRE 2024 Restore Any Image Model (RAIM) in the Wild Challenge (2024)

2.2.1 Paired Data with R-GT

To facilitate the model validation, we first provide some paired data in the following scenarios, where both the input low-quality image and the high-quality R-GT can be collected. Examples can be found in Figure1.

Image denoising. Compromising with the size and cost, the photosensitivity of imaging sensors, especially on mobile phone products, is limited. Meanwhile, the illumination of shooting scenes can be poor, especially in low-light imaging. Image denoising is a very fundamental requirement for image restoration.

Image super-resolution. The focal length of mobile phone cameras is limited, making it hard to meet the needs of continuous and ultra-long magnification zoom. Therefore, mobile cameras will be equipped with digital zoom algorithms, namely super-resolution algorithms.Out-of-focus restoration. The autofocus (AF) algorithm cannot guarantee 100% focus accuracy. In fleeting moments of excitement, such as blowing birthday candles, fireworks, etc, the image restoration algorithms are expected to remedy slight defocusing shots.

Motion deblur. Due to the limitations in aperture size and sensor capability, mobile phone cameras face a trade-off between shutter time and motion blur. A longer shutter may enhance the noise reduction performance, but it is prone to motion blur when encountering foreground object motion or handheld motion. An effective motion deblur algorithm is demanded.

The combinations of the above. When capturing a real-world photo in the wild, the above issues are usually triggered simultaneously by several factors, such as motion blur in high magnification super-resolution, out-of-focus in low light environments, etc. When multiple problems appear in the image, a strong model is needed to solve them jointly.In this challenge, we use these data to calculate the full-reference metrics to partially measure the effectiveness of the algorithms and screen the top performers in the early stage.

2.2.2 Data without R-GT

In many practical scenarios, the R-GT is very difficult to collect, and the image restoration performance is hard, if not possible, to be measured by full-reference metrics. In this challenge, we also provide the data with the following commonly encountered issues in practice. Examples can be found in Figure2.

Smoothed details and textures. Limited by hardware and on-chip computing power, images captured by mobile phone cameras often face a trade-off between noise/artifacts reduction and details/texture richness, impacting the visual quality. However, due to the lack of effective quantitative measures, evaluation can only be done through subjective observation.

Text stroke adhesion in super-resolution. In the telephoto mode (e.g., equivalent focal length larger than 230mm), shooting small text from a distance is an important yet highly challenging task. Text stroke adhesion, or super-resolution errors (i.e., presenting wrong characters), will greatly deteriorate the user experience.

High light edge and color artifacts. The optical system of mobile phones is limited, and prone to purple edges, green edges, halos, and fake textures in high-light areas. This problem occurs frequently in reflective scenes, backlight scenes, night scenes, etc., which greatly affects the user’s perception.

Low-frequency color noise/blocks/bandings. The low SNR of the input in mobile phone cameras demands a heavy denoising algorithm to output a clean image. However, due to factors such as computing power and storage, the bit-width of the ISP system is limited. When transitioning from the linear domain to the nonlinear domain, visual color noise, blocks and bandings often appear.

High-frequency aliasing and Moire pattern. Due to the resolution of imaging sensors, the Moire pattern can appear at specific distances and frequencies. Although users have certain expectations (or understanding) about the appearance of Moire patterns, they still hope to reduce the probability and severity of Moire patterns without affecting the clarity of the image.

2.3.1 Quantitative Measure

$SCORE=20\times\frac{PSNR}{50}+15\times\frac{SSIM-0.5}{0.5}+20\times\frac{1-$.

2.3.2 Subjective Evaluation

For the test data without R-GT, we judged the perceptual quality of the restored results by visual inspection. Specifically, we invited 18 experienced practitioners and conducted a comprehensive user study. The following features were considered in the evaluation:

Textures and details. The restored image should have fine and natural textures and details.

Noise. Noise, especially color noise, should be eliminated. Some luminance noise can be kept to avoid over-smoothness in flat areas.

Artifacts. Various artifacts, such as worm-like artifacts, color blocks, bandings, over-sharpening, and so on, should be reduced as much as possible.

Fidelity. The restored image should be loyal to the given input.More details have been discussed during the competition with all participants by referencing specific images and model outputs.

2.4.1 Phase 1: Model Design and Tuning

In this phase, participants can analyze the given data and tune their models accordingly. We provided:

2.4.2 Phase 2: Online Feedback

In this phase, participants can upload their results and get official feedback. We provide:

Only the low-quality input images are provided, and the participants can upload the restoration results to the server and get the quantitative scores online. Users can also upload their results of the images without the R-GT provided in Phase 1 to seek feedback. The organizers will provide feedback to a couple of teams that get the highest quantitative scores of the images with R-GT.

2.4.3 Phase 3: Final Evaluation

In this phase, we provide:

In this phase, we select the top ten teams according to the quantitative score of the 100 images with R-GT in Phase 2, and then arrange a comprehensive user study on their results of the above 50 images without R-GT. The final ranks of the ten teams will be decided based on both the quantitative scores and the subjective user study, with the weight being 40% and 60%, respectively.

4.1.1 Generator model

As shown in Fig. 7, the model is based on the MWRCAN [13]. The model uses a UNet architecture that employs Harr wavelet transforms and inverse transforms for 2$\times$ downsampling and upsampling. The major convolution modules consist of $N$ Resblocks, where $N$ is 8 in this case. The channels of the Resblocks are marked in the diagram, and there is also a residual connection in each downsample or upsample block, they omitted these connections for the sake of diagrammatical clarity.

Self-attention in transformers enables the network to identify self-similar features throughout the entire image, thereby enhancing its semantic recognition capabilities. However, the attention structure becomes increasingly time-consuming as the feature size grows, rendering it impractical for high-resolution image restoration tasks. To strike a balance between performance and efficiency, the team integrated RESATT structures into the middle block of UNet. RESATT comprises $N$ basic blocks, each consisting of a res-block followed by a single-head self-attention block.

The UNET produces a 3-channel image called out1. To enhance the quality of the restored image, they incorporate a refinement module based on the EMVD[25] approach. This module helps to recover important details that may have been lost during the restoration process. The refinement module takes in the LR image and out1 as inputs and produces a single-channel fusion weight, denoted by $\alpha$. The final output image is obtained by blending the LR image and out1 using $\alpha$, i.e., $HR=(1-\alpha)LR+\alpha out1$. The refinement module is lightweight, comprising only three convolutional layers with a maximum of 16 channels. Despite its simplicity, it is capable of capturing details that are crucial for the final output image.

The team still insists on using GAN models for general restoration because they have found that diffusion models can lead to unacceptable distortions in text and regular textures. The model has approximately 341MB parameters and takes up 7GB of GPU memory and 180ms to infer a 512$\times$512$\times$3 image on a computer with a 4090GPU.

4.1.2 Image degradation

The official competition only provided 100 pairs of training data, as well as 200 images without ground truth in the validation/test phase. They found that the degradation level of the provided 100 pairs of training data is only consistent with 100 images in phase 2, which is relatively mild. The other images in Phase 2 and Phase 3 have a heavier blurring.

Based on the analysis presented, the team developed two GAN degradation models that introduce varying levels of blurring. They enlarge the generator in [28] by doubling the channels, as the degradation model. The first model was trained with the ESRGAN [40] training method and consisted of 100 pairs of training sets, with high-resolution images serving as input to the degradation GAN model and low-resolution images as ground truth. This model introduced a weak level of blurring.

For the second model, they fine-tuned the weak degradation model using the approach outlined in Ref [28]. They trained this model in an unpaired manner, using 50 high-blurring images from phase 2 as unpaired GT and 1000 high-resolution input images from similar scenes as unpaired input. This model introduced a higher level of blurring compared to the first model. When using the second degradation model, they utilize a human segmentation model and a text segmentation model to segment out the human images with heights <300 pixels and the text with heights <50 pixels. These segments are then replaced with the degradation results from the first degradation model. This strategy helps to reduce the gap between the input and ground truth for small human images and text, and the team has found that this trick improves the fidelity of the results in these regions.

4.1.3 GAN training

The team has an internal ultra-high-definition dataset consisting of approximately 10,000 images. The main scenes include common animals and plants, Chinese and English text, as well as some common urban and rural scenes, which can cover the typical shooting scenarios of mobile phones. They used the two aforementioned degraded GAN models to degrade these images, resulting in a dataset of 20,000 training pairs.

To develop a high-quality image restoration model for phase 2 quantitative measures, they utilized a GAN model trained on 10,000 degraded training pairs from the initial degradation model. The Generator’s learning rate was set to 1e-5, with a batch size of 24 and a patch size of 512. The team began training with only L2 loss for $\sim$10,000 iterations, then fixed the loss to include $L2+1*PerceptualLoss+0.1*GANLoss$ for an additional 140,000 iterations. They then fine-tuned the model for $\sim$20,000 iterations with $L2+0.1*PerceptualLoss+0.01*GANLoss+4*LPIPS$ and a lower learning rate of 1e-6 on the official training set (100 pairs) to achieve a slightly higher quantitative score. The discriminator setting is the same as RealESRGAN [39].

For phase 3, the team continued fine-tuning the model for approximately 100,000 iterations using $loss=L2+0.1*PerceptualLoss+0.01*GANLoss+4*LPIPS$, with a learning rate of $1e-5$. They used a mixed dataset with $80\%$ strong degradation and $20\%$ weak degradation by adjusting the training file list ratio. Finally, they crop each training image into $512\times 512$ patches and select the top 10 patches with the higher NIQE score for each image. They continued fine-tuning the model on this subset with a learning rate of $1e-6$ for $\sim$50,000 iterations. Higher NIQE patches generally have richer textures and they found that fine-tuning the model on this subset resulted in better image details.

4.2.1 Detailed Method Description for Phase2

The dataset they used is LSDIR[17].During training, they construct pairs with a resolution of 128x128. The degradation hyperparameters are the same as those for real-esrgan.They trained 92k iterations with batch size=12 (3 for one GPU, total 4 GPUs) in stage-1 and Adam’s learning rate is 1e-4.

In the second stage of training, the team added adversarial loss and perceptual loss, and instead of using lsdir, they only used 100 paired images provided by the official competition. The results show that the degradation distribution of the official evaluation data is close to that of the 100 images.The specific loss function coefficients are shown below. They trained for a total of 140k iterations in the second stage, with a batch size of 12. The learning rate for Adam is 5e-5.

There is nothing special about the test. For an image, just input it directly into the trained model.

4.2.2 Overall Approach

In recent years, the diffusion method has achieved remarkable results in the field of image generation, and many methods have recently explored its application in the field of image restoration. Due to the unavailability of data for phase 3 of this competition, the distribution of degradation may differ from phase 2. To increase the generalization ability of our solution, the team use SUPIR[49] as our baseline model.

SUPIR is trained on 20 million images and has good modeling of the distribution of natural images. It supports multiple parameters such as positive prompt, negative prompt, and Classifier free guidance scale to adjust the enhanced results.Due to the short competition time and the lack of open-source training code for SUPIR, they did not perform any training fine-tuning on SUPIR, but based on its RGB results. To obtain preliminary RGB results, most official default configurations have not been changed. Only the parameters listed in the table 2 are different from the default parameters.

Although the results generated by diffusion can be natural in most scenes, fidelity issues may arise in some small texture scenes, such as text, patterns, and architectural lines.Especially in the field of photography, this distortion may be unacceptable to professionals, and even worse than not being processed.To alleviate this issue, as shown in figure 8, they will perform another fusion process based on the SUPIR results to obtain the final result. The input of the fusion module includes the SUPIR result, the original image, and a 0/1 mask. To obtain this 0/1 mask, they used the DeSRA[46] method. For the sake of fidelity, the fusion module will perform a lighter enhancement on the area with a value of 1 in the mask (e.g., using GAN-based methods), while the area with a value of 0 will be kept as unchanged as possible(i.e., using SUPIR’s result).They introduce our fusion module and DeSRA method in sections 4.2.3 and 4.2.6 in detail, respectively.

4.2.3 Fusion Network

4.2.4 Architecture

To ensure the authenticity of the results from diffusion-based models, their fusion module performs fine-tuning based on a binary mask. Specifically, the model takes in three components during inference: the output from SUPIR, the original image, and a binary mask. Areas, where the mask is zero, indicate that the results from SUPIR are already optimal and do not necessitate any modifications, so they will keep this area. Conversely, regions where the mask is one suggest that the results require re-generation to maintain fidelity. They will replace this area with the corresponding LR part to input the model.

In light of the above, the fusion module operates akin to an image inpainting task[53, 54], with the key difference that the masked areas are not entirely devoid of information; instead, they contain low-quality images that are awaiting enhancement.In the training process, the team continue to follow the Real-ESRGAN strategy to generate paired (LR, GT) on the LSDIR dataset.As illustrated in the figure9, their model backbone continues to employ SRFormer[59] (consistent with Phase 2), with the only change being the inputs. At this point, the input will encompass the LR, mask, as well as the GT and LR combinations derived from the mask. In the inference process, the GT depicted in Figure9 should be substituted with the outcomes yielded by SUPIR.

For the mask used during training, they generate it randomly following the method outlined in STTN[53], while during testing, they utilize the DeSRA[46] approach to obtain the mask. Regarding the DeSRA method, it will be introduced later.

Given that the inputs already contain regions of high quality, the loss function must be correspondingly modified to account for this.Similar to image inpainting tasks[54], the loss function encompasses hole loss, valid loss, perceptual loss, and adversarial loss. Notably, for the generated fake image, the discriminator employs the technique of using soft labels when calculating the least square loss[54], rather than the hard labels of 0 and 1. This design allows the discriminator to better discern potential mask areas.

4.2.5 Training Details

The dataset they used is LSDIR[17].During training, the team constructed pairs with a resolution of 144x144. The degradation hyperparameters are the same as those for real-esrgan.They trained 172k iterations with batch size=8 (2 for one GPU, total 4 GPUs) and Adam’s learning rate is 1e-4.The specific loss function coefficients are shown below.

The generation of random masks during training can be referenced at the specified line in the following GitHub repository: STTN GitHub Repository.

4.2.6 DeSRA Method

With the fusion model in place, it is necessary to ascertain the masks used during testing, specifically identifying the regions where the diffusion results are distorted.A straightforward method involves manual annotation of masks, but this approach is not only unfair in the context of competition but also labor-intensive.

The team employ the methodology from DeSRA[46] for identifying GAN artifacts, utilizing a combination of structural similarity metrics and semantic segmentation outcomes to generate masks.To be precise, they ascertain the mask by contrasting the outputs from the GAN model with those from the diffusion model. The GAN model utilized in this process is the one that has been adequately trained during Phase 2. This choice is motivated by the fact that, despite the GAN model’s potential shortcomings in visual quality, it excels in preserving fidelity in intricate details such as text and textures.By adjusting the parameters, the team strives to align the distribution of the masks with human visual perception. It is important to note that no special parameters are used for any individual image; the same set of parameters is applied consistently across all 50 images.

To enhance the accuracy of the segmentation, they have utilized the Mask2Former model[10] for this task. Compared to the SegFormer model[45] used in the original DeSRA, Mask2Former represents a more advanced approach.Within the provided code, they have included scripts for mask generation, which encompass all the parameters used, including the weights for semantic categories, contrast_threshold, area_threshold, and so on.

4.7.1 Phase III: Evaluation on Subjective Measurements

4.9.1 Methods details

The team observed that the input images and evaluation criteria of the two phases are different. The input images in phase 2 have higher quality. The evaluation criteria for this phase are based on reference evaluation indicators. The input quality of phase 3 is relatively low, with a more serious blur. This phase uses the method of manual scoring to select images with better visual effects as the winners. Taking into account the existing solutions, the team decided to adopt a gan-based approach in phase 2 to obtain higher objective indicator scores. In phase 3, a diffusion-based approach is adopted to make the results more visually appealing.

4.9.2 Phase 2

The organizers provided 100 pairs of training images whose input quality and imaging style are similar to the test set of the first stage. Therefore, they chose DiffIR [44] for this stage. It only uses the diffusion process to model the condition branch, and the main network is trained using the GAN loss, so it rarely destroys local details (such as text, small faces), and can obtain a higher objective evaluation index. They directly use the pre-trained model of DiffIR and fine-tune it with the paired dataset provided by the organizer, so that the team can quickly obtain a good result. The whole finetuning process continues 6.6 k iterations with a batch size of 48. In addition, in the process of preparing the dataset, they adopted a multi-scale downsampling strategy, hoping that the model could gain knowledge of different scales. The downsampling scales are set to 0.75, 0.5, and 0.33, respectively.

4.9.3 Phase 3

In phase 3, the test set provided by the organizer has a significant domain gap from the test set of phase 2, and the degradation is more severe. The team thinks they cannot directly use the model that performs well in phase 2 to obtain good visual results in the second stage, so they switched to using the methods [36, 48, 43, 34] of the pre-training diffusion model. As shown in Fig. 15, the team chooses the popular ControlNet [56] as the solution. Following [19], they use pretrained VAE encoder as the image encoder. In terms of training data, they choose LSDIR [17], which contains tens of thousands of texture-rich images. As for data degradation, to match the more severe degradation of the test set, they choose realesrgan’s [41] degradation pipeline to synthesize paired data. They train the model with a batch size of 32 for 100k iterations. In the inference stage, the team resizes the input to 2048 before feeding it into the model, which aims to preserve small structures like texts as shown in Fig. 16. The team also adopts the LRE strategy proposed in [43] to improve fidelity. The pre-trained diffusion model in this solution is SD2-base [32].