MasonFace wrote: ↑Sun Dec 23, 2018 7:45 pm
I was tinkering around with ESRGAN and SFTGAN this morning and found a workflow that produces consistently
decent results.
My new method involves appying a 1 pixel guassian blur to the vanilla texture then feeding it into ESRGAN, then applying a sharpen filter to the result, then upscaling that with SFTGAN. The results look a little better to my eye:
In my experiments with waifu2x, I found that blur does produce better results with certain edges and curves but the output image remains visibly blurred, which is not a good thing.
I found a workaround by "softening" the source image instead. First I take the original image and scale it up by 4x using the
xBRZ Scaler Test with either xBR or xBRZ scalers, then open it in GIMP and apply the pixellate filter at 4 pixel radius, then resize the image to the original dimensions with the nearest neighbour method. This only slightly blurs the contrasts that are inevitable in an 8-bit image, which is enough for waifu2x to process the image in a more proper way.
Here's a comparison of a sample DF texture. Raw image scaled to 4x (TTA2) on the left, softened on the right (both with RGB mode scaling):
I wasn't content with the fact that the result is still pretty blurry, so I tried blending several layers using waifu2x processing with two passes:
In the above, to the left is the original texture scaled to 4x with xBR, to the right a blend (using G'MIC plugin for GIMP, Blend [median]) of the same xBR thing with two versions of waifu2x-processed image that was smoothed: RGB scaling with two passes and y-mode scaling with two passes. The image is also converted to the original palette using mtPaint with Scattered (effect) dithering method).
I described the blending process in a bit more detail
here.
And here's a test of the dog (4x the original size but then increased by 2x using nearest neighbour for better visibility):
This is a blend of three layers, all of which were produced by waifu2x from a "softened" image (as described above): 4x Y-mode (TTA2, one pass), 4x Y-mode (TTA2, to passes), and 4x RBG mode (TTA2, two passes). The result was converted to the original palette with mtPaint using Scattered (effect) dithering method
I wonder what results you might get with ESRGAN or SFTGAN if you use the softening method instead of Gaussian blur.
UPD: Here's the horse sprite, processed in almost the same way as the dog except that the one-pass layer out of the three was with RGB mode instead of Y-mode:
You might have noticed that the "xBRZ softening" produces some very good results on the sprite edges compared to the raw image.