Text from the interactive explanation
Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances
Brian B. Moser, Federico Raue, Stanislav Frolov, Sebastian Palacio, Jörn Hees, Andreas Dengel
Original paperA hitchhiker’s tour of super-resolution
On TV, a detective squints at blurry footage and says “Enhance!” — and a sharp face appears. A real small image can’t hold detail that was never recorded. So what is a super-resolution model actually doing?
Low-res pixels changed: 0 of 64
Hidden values per kept pixel: 16
Valid pasts explored: 1
Drag a tower up or down (or tap one, then use Raise / Lower; with the keyboard, focus the scene and use the arrow keys and + / −). The other 15 towers under its slab shift to compensate, so the slab never moves.
Each glass slab is one pixel of a small picture: the average brightness of the 16 towers beneath it. The towers below keep rearranging into different sharp images, and every arrangement averages to exactly the same slabs. One small image, many sharp pasts.
That is the problem at the heart of the survey “Hitchhiker’s Guide to Super-Resolution” by Moser, Raue, Frolov, Palacio, Hees and Dengel (IEEE TPAMI, 2023). Super-resolution (SR) enhances low-resolution (LR) images to high-resolution (HR) ones, for uses that range from everyday photos to satellite and medical imaging. It stays hard because it is “notoriously ill-posed: several HR images can be valid for any given LR image.”
The survey sets out to guide newcomers through “the crowded field of SR”: how guesses are scored, how models are trained and built, and where research is heading. This tour follows the same route with one question in hand: if a blurry image fits countless sharp ones, what is an SR model really doing, and how would we know it did well?
Shrinking is easy
The survey writes the link between a sharp image y and its small version x as a degradation mapping:
In words: small picture = Degrade(big picture; settings). The settings δ hold the scale factor s and extras such as the type of blur.
In practice D “is often unknown and therefore modeled, e.g., with bicubic downsampling”, and benchmark LR images are typically made with bicubic interpolation plus anti-aliasing. On this page we use the simplest possible D: average every s × s block of pixels into one pixel , on brightness only. (The first deep SR model, SRCNN, also worked on the brightness channel only.)
All images here are procedurally generated test images: a zebra hide (a nod to the zebra in the classic Set14 benchmark), a butterfly wing (Set5 has a butterfly), an eye chart and a brick wall.
Hover over or tap a pixel of the small image to see the numbers that were merged into it.
Shrink an image
Going this way is easy. Going back means inventing s² − 1 numbers for every pixel you kept. Which numbers?
Which one is real?
Below is one small image and four sharp candidates. Exactly one of them is the original that we shrank. Tap the one you think produced the small image.
Predict, then check
How can four different images all be “the” answer? Our D only keeps the average of each 4 × 4 block. Any change that adds as much brightness to some pixels of a block as it removes from others leaves every average, and therefore the small image, untouched. The candidates differ only in such changes: we borrowed fine detail from other images (or from a shifted copy of this one) and re-balanced it inside every block.
The survey puts it plainly: inverting D “is ill-posed because one LR image can lead to multiple nonidentical HR images.” Your eye may still favour the most natural-looking candidate. That preference is knowledge about what images usually look like, and it is exactly what an SR model has to learn.
So SR can’t recover the past; it has to choose one. The oldest way to choose is a fixed rule.
Guess by rule
Interpolation fills the gaps between known pixels with a fixed formula. The survey describes three classics:
Nearest neighbour copies the closest pixel. Fast, but “there are no interpolated transitions, which results in blocky artifacts.”
Bilinear blends the 2 × 2 nearest pixels, giving smoother transitions.
Bicubic fits smooth curves through a 4 × 4 neighbourhood. It “delivers much smoother results” but is the slowest of the three. It is still the usual choice when a deep model starts from an interpolated image.
Drag the divider to compare the guess (left) with the truth (right). The two scores underneath, PSNR and SSIM, are computed live against the truth.
One row, three rules
Upscale ×4 by rule
Smooth isn’t sharp. Bicubic usually scores best of the three, yet it still looks soft. We just used two scores. What do they actually measure?
Be the judge: your eyes vs the numbers
Three quick rounds. Each shows the truth and two candidates. Tap the candidate that looks more like the truth. Then see what two standard scores say.
Round 1 of 3
Your votes are a Mean Opinion Score with a single rater. MOS asks human viewers to rate images, typically from 1 (bad) to 5 (good), and averages the ratings. The survey calls subjective evaluation the most appropriate way to assess images, but gathering raters is slow and costly.
PSNR (peak signal-to-noise ratio) compares pixels one by one: it is 10·log₁₀(255² / mean squared error), in decibels, higher is better. The survey warns that it “focuses on pixel-level differences instead of mammalian visual perception”, and that “slight changes in pixels (e.g., shifting) can lead to a significantly decreased PSNR, while humans barely recognize the difference.”
SSIM (structural similarity) compares luminance (mean), contrast (spread) and structure (correlation) in small windows; 1 means identical. We compute both scores exactly as the survey’s equations 5–12 (SSIM averaged over 7 × 7 windows).
Newer learned metrics such as LPIPS compare features extracted by a neural network instead of raw pixels; its authors showed that it “aligns better with human judgments than PSNR or SSIM”, though its quality depends on the feature extractor underneath.
The scores are cheap and useful, but they aren’t eyes. Keep that in mind: models are trained to push scores up. Time to watch one learn.
Guess by learning: train a tiny super-resolver
Instead of a fixed rule, let a model learn what detail usually hides inside a blurry patch. The recipe in the survey: take sharp images, generate small versions of them, and treat the originals as the answers (“generating LR pairs and treating original samples as the HR version”).
Training lab
The model. For every pixel of the small image, a tiny network reads its 5 × 5 neighbourhood and outputs the s × s sharp pixels underneath it. That is a small convolutional network ending in a sub-pixel (“pixel shuffle”) layer, the upsampling trick introduced with ESPCN. It has one hidden layer of 20 units: 604 numbers to learn. It trains on five procedurally generated images and is tested on the held-out butterfly wing.
Residual learning. VDSR’s trick was a skip connection from the interpolated input to the output, so the network only adds the missing high-frequency detail on top of bicubic. The survey notes this gives “a target distribution that is normal and reduces the learning difficulty immensely.” The histograms come from the actual training data on this page.
Loss. L1 averages absolute errors; L2 averages squared errors, so it punishes big mistakes more. The survey notes L1 is more common in the literature because L2 “is very reactive to extraordinary values.”
Zero-shot. ZSSR trains on the one image at hand: it shrinks the small image even further, learns to undo that, then applies the network to the small image itself. The survey calls this “highly practical, especially for applications where application-specific datasets are rare or non-existent.” Try it at ×4: our 32 × 32 test image shrinks to just 8 × 8 for training.
This is a toy model running in your browser, a few hundred numbers trained for seconds. Real SR networks have thousands to millions of parameters (see the scoreboard below) and train on thousands of photographs, so don’t read its decibels as a benchmark.
Notice what “better” means here: a higher PSNR. But a pixel-wise loss rewards a cautious guess. Why does that produce blur?
Why careful learners paint blur
Back to the towers. Same wing, same slabs, and a set of sharp pasts that all fit them. If a model has to commit to one image but cannot know which past is true, what single guess keeps its pixel error lowest?
Average the pasts
The pixel-wise average of all plausible pasts is the single guess with the smallest expected squared error. It scores the best PSNR against the true past, whichever past that turns out to be. But averaging sharp, disagreeing details cancels them out. The survey observes the same thing in real models: pixel losses “favor a high PSNR”, and “the resultant images tend to be blurry.”
The right-hand map shows where the pasts disagree. It is dark wherever the wing is smooth and bright along edges and veins. That pattern underlies the uncertainty-driven loss (UDL) of Ning et al., discussed in the survey: since “the uncertainty is sparse for SISR due to many smooth areas”, training should prioritise “texture and edge pixels that are visually more significant than pixels in smooth regions.”
Toy illustration: these pasts are built by giving each block the same amount of fine detail as the original, borrowed from other images and re-balanced to keep every block average. Real image statistics are richer, but the arithmetic of averaging is the same.
To look real, a model must commit to one sharp past instead of blending them all. That is what GANs and diffusion models do.
Commit to one sharp past: GANs and diffusion
GANs. A generative adversarial network pits two networks against each other. The generator is the SR model; a discriminator “tries to distinguish between samples coming from the generator and samples from the actual dataset.” To fool it, the generator must produce detail that looks real rather than a safe average. Such models are often paired with content or texture losses that compare feature maps of a pretrained network instead of raw pixels. The survey notes SR models generally perform better with an adversarial loss, but training stability is hard to reach (for example “mode collapse”).
Diffusion. Denoising diffusion models such as SR3 (“Super-Resolution via Repeated Refinement”) start from pure noise and remove a little of it at a time, guided by the small image. The forward direction just mixes an image with Gaussian noise; the survey’s Eq. 34 writes it as
Slide toward 0: the image dissolves into noise.
A network is trained to predict the added noise, which lets each refinement step (Eqs. 36–38) undo a little of it. Run the reverse process and the sampler drifts toward one sharp past, a different one each time. The survey reports that SR3 “generates sharp images with more details than its regression-based counterparts”, a result the SR3 authors also checked with 50 human subjects. It also flags a caveat: SR3 “seems prone to bias”, e.g., on faces, “too smooth skin texture by dropping moles, pimples, and piercings.”
Reverse refinement
Toy sampler with an exact denoiser over a handful of pasts that all fit the same small image. It uses the survey’s refinement step (Eq. 38) with a 50-step noise schedule. SR3 learns this denoiser with a neural network trained on many images. Past F carries a small dark dot, like a mole. Remove it from the set and the dot can never appear: an illustration that a sampler only produces detail it knows, not a reproduction of SR3’s bias.
Different philosophies, one decade of progress. How have all these ideas added up on the benchmark the survey tabulates?
A decade on the scoreboard
These numbers are not computed here. They are PSNR and SSIM on the Set5 test images, as collected in the survey’s Table I. Each extra decibel is hard-won: +3 dB means the average squared error has roughly halved.
Set5 benchmark (Table I)
Survey caveat: the ordering of Table I “reflects roughly the performance ranking. However, some comparisons are not fair due to different settings, e.g., NAS versus unsupervised results.” For example, CAR learns the downscaling jointly with the SR model. Years are as given in the survey (Table I/text and supplementary Table 4, which differ for a few models).
Inside the models
Learned upsampling with a transposed convolution spreads each input value over a k × k window, stepping 2 pixels at a time. When k isn’t divisible by the stride, some output positions are visited more often than others (supplementary Fig. 23).
Back to “Enhance!”
So what does “Enhance!” really do? Super-resolution doesn’t recover the lost past; it chooses a plausible one , using what it learned about how images usually look. A decade of progress has meant choosing better and faster, and learning to judge “better” more the way people see.
The survey closes with open problems, among them:
Scores that match our eyes
Zoom by any amount
Learning without pairs
Better learning objectives
Lighter models, clearer recipes
Source & credits
The article is © 2023 The Authors and licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Benchmark numbers are reproduced from Table I and supplementary Table 4 of the survey and restyled as interactive charts; family colours follow the survey’s architecture groups. Quotations are from the survey.
All images on this page are procedurally generated. Lab numbers (PSNR, SSIM, training curves, differences) are computed live in your browser. The training and diffusion labs are simplified toy models built for explanation, not the survey’s models, and the “many pasts” are illustrations of a simplified box-averaging degradation.
Topics in this iPaper
Computer science
Artificial intelligence, computer science, deep learning
Abstract and Index Terms, page 1 (p. 9862)
super resolution
is the process of enhancing Low-Resolution (LR) images to High-Resolution (HR)
Section I (Introduction), page 1 (p. 9862)
deep learning
we, in contrast, concentrate exclusively on DL methods given their proven superiority
Section I (Introduction), page 1 (p. 9862)
image quality assessment
fall under the umbrella term Image Quality Assessment (IQA)
Section II-B, page 2 (p. 9863)
generative adversarial networks
The core idea is to use two distinct networks: a generator G and a discriminator D
Section III-B, page 5 (p. 9866)
denoising diffusion probabilistic models
Denoising Diffusion Probabilistic Models (DDPMs) [62] exploit this insight
Section III-C, page 6 (p. 9867)
neural architecture search
Recently, a new field called Neural Architecture Search (NAS) has gained popularity
Section IX, page 16 (p. 9877)