AI Upscaled Textures

[MasonFace]

Think these are by far the best results on these types of NPC sprites.

Besides the xBRZ upscaling part, are you able to batch this process? If not, once you're happy with your results, document precisely what the steps and parameters for each step are and I will batch them all collectively. Any steps prior to xBRZ I can batch, then we can split up the workload of manually upscaling each sprite with xBRZ, then I can batch the remaining steps post-xBRZ upscaling, then run them all through whichever ESRGAN model interpolation you decide works best. I'll return the results back to you if you want to attempt any post-processing.

I'm curious if I can squeeze out any extra detail by compositing your results with Gigapixel results. I may try that tonight.

[phredreeke]

Any steps prior to xBRZ I can batch, then we can split up the workload of manually upscaling each sprite with xBRZ,

Code: Select all

md xbrz4
for %%f in (*.png) do ("F:\bloodupscale\ScalerTest\ScalerTest.exe" -4xBRZ %%f xbrz4\%%~nf.png )

Replace F:\bloodupscale\ScalerTest\ with the path to ScalerTest of course.

Edit: the final batch of masks finished processing http://s000.tinyupload.com/index.php?fi ... 4528043167

[MasonFace]

Ah, very good! That will speed things along very nicely! Thanks again Phredreeke for the masks and the .bat code!

[MrFlibble]

Besides the xBRZ upscaling part, are you able to batch this process? If not, once you're happy with your results, document precisely what the steps and parameters for each step are and I will batch them all collectively.

Honestly I've never given a thought about batching. When I was still using waifu2x my basic idea was to collect all sprites into a single sheet (keeping enough space between the sprites of course to prevent them from merging together) and then do all manipulations to that image. If there is a programme that stitches together multiple images (or a GIMP function that tiles layers) that would aid this kind of process immensely.

But I see Phredreeke has made a very clever batch file to automate the xBRZ process, that's very cool! Thanks!!! ^_^

Here are all the steps I have used to get the results above:

1) The sprites are set against a solid dark blue background same as here.

2) Pre-processing in GIMP:

• Shift+O to switch to select by colour, click on background to select it
• right-click selection, choose Select -> Grow, and grow selection by 1 pixel; copy selected area, deselect (Shift+Ctrl+A)
• increase image size to 400% using Sinc (or whatever its equivalent in v2.10; actually my v2.8 has a bug and the labels for Sinc and Bicubic are swapped for whichever reason)
• decrease image size to 25% (i.e., original resolution) w/ Bicubic interpolation
• paste the selection copied earlier (this is to keep the anti-aliasing effects from resizing off the sprites' edges)
• select Filters -> Enhance -> Antialiase
• export image

3) Pre-processing with xBRZ:

• scale image up 4x with xBRZ
• load the result in GIMP, apply Gaussian blur at 1.5 pixel radius
• scale down to original dimensions with Bicubic interpolation

4) Scale result with ESRGAN/Manga109 interpolated model at alpha = 0.2

5) Get the mask with GIMP & mtPaint:

• open the neurally upscaled image in mtPaint and save a copy (e.g. with a "KN" or "mask" prefix)
• apply Kuwahara-Nagao blur at 2 pixel radius with the Protect details option on
• convert to 8-bit palette:
  • load the intended paletted image; it may be the original sprite or a generic image containing the Daggerfall palette but it must contain the extra dark blue colour we're using for the background
  • select Image -> Convert to Indexed. Select current palette and no dithering. Select options and change colour space to RGB (default is sRGB). Click OK to finish conversion. Save and close
• open both the original upscaled image and the Kuwahara-Nagao indexed one in GIMP
• select background in the KN image (select by colour), grow by one pixel, and copy the selected area to the original upscale; save/export the resulting image
• optionally you can convert the result to the 8-bit palette as well using the steps above

This should give you clean edges against solid, single colour background that you can at any time alter or delete altogether.

I briefly checked mtPaint documentation but I'm not sure if there's anything to automate the above processes.

[MasonFace]

Here is the result of your sprites composited with Gigapixel results:

pTsM6Qk_Gigapixel60_composite.png (88.66 KiB) Viewed 3762 times

And below is the inn keeper with a quickly touched up face.

InnKeepTouchedUp.png (32.14 KiB) Viewed 3762 times

[phredreeke]

Honestly I've never given a thought about batching. When I was still using waifu2x my basic idea was to collect all sprites into a single sheet (keeping enough space between the sprites of course to prevent them from merging together) and then do all manipulations to that image. If there is a programme that stitches together multiple images (or a GIMP function that tiles layers) that would aid this kind of process immensely.

I use ImageMagick to combine upscales and masks

Code: Select all

md composite
for %%f in (*.png) do ("C:\Program Files\ImageMagick-7.0.7-Q8\Magick.exe" montage %%f "..\proc-downscalepalette2\%%f" -tile x1 -geometry +0+0 png32:composite\%%f)

[MrFlibble]

Here is the result of your sprites composited with Gigapixel results:

Interesting! Gigapixel seems to produce less cartoonish faces overall. I suspect it does have a semantic level somewhere inside which tries to identify parts of the image and then enhance them accordingly. But I cannot be certain since I've only seen some output from its processes and never did any testing of my own.

The girl looks a lot older though OTOH, very nice job on the hair.

And below is the inn keeper with a quickly touched up face.

I think it's very cool!

Could you try the small NPC flats with Gigapixel too? I wonder how it will handle their faces.

I use ImageMagick to combine upscales and masks

Code: Select all

md composite
for %%f in (*.png) do ("C:\Program Files\ImageMagick-7.0.7-Q8\Magick.exe" montage %%f "..\proc-downscalepalette2\%%f" -tile x1 -geometry +0+0 png32:composite\%%f)

Great, thanks for sharing this!!!

[MasonFace]

I'm going to take a look at this new SC-FEGAN neural network this weekend. It looks really cool! Could help this project out with restoring the faces of the NPCs and maybe the portraits, too.

https://github.com/JoYoungjoo/SC-FEGAN

An example of what it can do with incomplete pictures of faces:

restoration.jpg (945.08 KiB) Viewed 3707 times

Of course it was trained on actual photographs, so the results may be too realistic. Perhaps we can run some filters on the results to make it blend in a little better. Kind of a long shot, but could be interesting nonetheless.

[Alyndiar]

I'm going to take a look at this new SC-FEGAN neural network this weekend. It looks really cool! Could help this project out with restoring the faces of the NPCs and maybe the portraits, too.

https://github.com/JoYoungjoo/SC-FEGAN

An example of what it can do with incomplete pictures of faces:

restoration.jpg

Of course it was trained on actual photographs, so the results may be too realistic. Perhaps we can run some filters on the results to make it blend in a little better. Kind of a long shot, but could be interesting nonetheless.

Wow! If anyone can help me set this up on my side, I'd love to play with this! I just have no idea on how to make this work. But this is just amazing...

[MasonFace]

I'm not sure where to even begin to tell you how to get this running, but I can tell you how I got it working for me.

This is a scholarly project, so they tend to be focused more on results than making it easy for the uninitiated to install and run. You'll have to jump through several hoops to get it up and going.

I'm running Windows 10, so if you're using another operating system the instructions may be slightly different.

Important note: Artificial Intelligence algorithms are usually ran on the GPU because these "tensor type" systems are much more easily solved on GPU architecture. As of time of writing this, only NVIDIA graphics cards have the accompanying software to allow the AI to run on the GPU. This software is called CUDA and you will likely need to install this to speed up the calculations. If you don't have an NVIDIA graphics card, then you will have to run it on your CPU. Even the fastest CPUs are an order of magnitude slower than GPU for these calculations, so expect it to be slow. The authors don't specifically mention any of this, so it is possible that it somehow doesn't require CUDA to run on the GPU, but I believe it does.

The first thing you'll want to do is install Python 3: https://www.youtube.com/watch?v=3lGhtIqT0Tk

Now that you have Python installed, PIP should have been installed with it which you'll need to download all the other dependent packages.

Go to the SC-FEGAN Github page and look at the items in the Dependencies section:
https://github.com/JoYoungjoo/SC-FEGAN

Open command prompt on your computer. This is where we will enter our terminal commands. If you don't have a shortcut for command prompt saved, you can find it using the search bar next to your start menu at the bottom left corner of your desktop and searching for "CMD".

Inside of command prompt, use PIP to install each of the dependent packages that are on the dependencies list. For example, to install tensorflow, type "PIP install tensorflow" and press enter. Repeat this step for each item in the dependencies list.

Next, you will need to clone their Github repository. The easiest way is with Github desktop, but if you don't want to install that, you should be able to just download the repository by clicking on the green "Clone or downlaod" button near the top right of their Github page. Use the "Download ZIP" option. Extract the contents of the zip file wherever you like. Open the "SC_FEGAN" folder you just extracted and if it doesn't have a folder in there named ckpt, then create one. I don't know why they don't mention that in their instructions, but it almost threw me for a loop.

Next you will need to download their model. This contains all the trained weights that governs how the AI reacts to inputs. On their Github page, right beneath the Dependencies table, there is a link to the model that's hosted on their Google Drive account. Go there and download "SC-FEGAN.ckpt.data-00000-of-00001" into your SC_FEGAN/ckpt folder.

If you don't have and NVIDIA graphics card that supports CUDA, I believe you will have to run the program on your CPU instead. You will have to modify the file "demo.yaml" using notepad, change "GPU_NUM 1" to "GPU_NUM" (basically just erase the number). Save and close the file and you're good to go.

Now open up command prompt again. Navigate to your SC_FEGAN folder using the change directory command (cd) then type "python demo.py" and hit enter. It should run from there.

pythonrundemo.PNG (8.86 KiB) Viewed 3629 times

If you have any trouble, just PM me so we don't clutter up the thread too much.