AI Upscaled Textures

Show off your mod creations or just a work in progress.
User avatar
MasonFace
Posts: 346
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Wed Mar 27, 2019 6:26 pm

Think these are by far the best results on these types of NPC sprites.

Besides the xBRZ upscaling part, are you able to batch this process? If not, once you're happy with your results, document precisely what the steps and parameters for each step are and I will batch them all collectively. Any steps prior to xBRZ I can batch, then we can split up the workload of manually upscaling each sprite with xBRZ, then I can batch the remaining steps post-xBRZ upscaling, then run them all through whichever ESRGAN model interpolation you decide works best. I'll return the results back to you if you want to attempt any post-processing.

I'm curious if I can squeeze out any extra detail by compositing your results with Gigapixel results. I may try that tonight.

phredreeke
Posts: 55
Joined: Mon Mar 11, 2019 9:13 am

Re: AI Upscaled Textures

Post by phredreeke » Wed Mar 27, 2019 6:45 pm

MasonFace wrote:
Wed Mar 27, 2019 6:26 pm
Any steps prior to xBRZ I can batch, then we can split up the workload of manually upscaling each sprite with xBRZ,

Code: Select all

md xbrz4
for %%f in (*.png) do ("F:\bloodupscale\ScalerTest\ScalerTest.exe" -4xBRZ %%f xbrz4\%%~nf.png )
Replace F:\bloodupscale\ScalerTest\ with the path to ScalerTest of course.

Edit: the final batch of masks finished processing http://s000.tinyupload.com/index.php?fi ... 4528043167

User avatar
MasonFace
Posts: 346
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Wed Mar 27, 2019 6:55 pm

Ah, very good! That will speed things along very nicely! Thanks again Phredreeke for the masks and the .bat code!

MrFlibble
Posts: 129
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Wed Mar 27, 2019 9:06 pm

MasonFace wrote:
Wed Mar 27, 2019 6:26 pm
Besides the xBRZ upscaling part, are you able to batch this process? If not, once you're happy with your results, document precisely what the steps and parameters for each step are and I will batch them all collectively.
Honestly I've never given a thought about batching. When I was still using waifu2x my basic idea was to collect all sprites into a single sheet (keeping enough space between the sprites of course to prevent them from merging together) and then do all manipulations to that image. If there is a programme that stitches together multiple images (or a GIMP function that tiles layers) that would aid this kind of process immensely.

But I see Phredreeke has made a very clever batch file to automate the xBRZ process, that's very cool! Thanks!!! ^_^

Here are all the steps I have used to get the results above:

1) The sprites are set against a solid dark blue background same as here.

2) Pre-processing in GIMP:
  • Shift+O to switch to select by colour, click on background to select it
  • right-click selection, choose Select -> Grow, and grow selection by 1 pixel; copy selected area, deselect (Shift+Ctrl+A)
  • increase image size to 400% using Sinc (or whatever its equivalent in v2.10; actually my v2.8 has a bug and the labels for Sinc and Bicubic are swapped for whichever reason)
  • decrease image size to 25% (i.e., original resolution) w/ Bicubic interpolation
  • paste the selection copied earlier (this is to keep the anti-aliasing effects from resizing off the sprites' edges)
  • select Filters -> Enhance -> Antialiase
  • export image
3) Pre-processing with xBRZ:
  • scale image up 4x with xBRZ
  • load the result in GIMP, apply Gaussian blur at 1.5 pixel radius
  • scale down to original dimensions with Bicubic interpolation
4) Scale result with ESRGAN/Manga109 interpolated model at alpha = 0.2

5) Get the mask with GIMP & mtPaint:
  • open the neurally upscaled image in mtPaint and save a copy (e.g. with a "KN" or "mask" prefix)
  • apply Kuwahara-Nagao blur at 2 pixel radius with the Protect details option on
  • convert to 8-bit palette:
    • load the intended paletted image; it may be the original sprite or a generic image containing the Daggerfall palette but it must contain the extra dark blue colour we're using for the background
    • select Image -> Convert to Indexed. Select current palette and no dithering. Select options and change colour space to RGB (default is sRGB). Click OK to finish conversion. Save and close
  • open both the original upscaled image and the Kuwahara-Nagao indexed one in GIMP
  • select background in the KN image (select by colour), grow by one pixel, and copy the selected area to the original upscale; save/export the resulting image
  • optionally you can convert the result to the 8-bit palette as well using the steps above
This should give you clean edges against solid, single colour background that you can at any time alter or delete altogether.

I briefly checked mtPaint documentation but I'm not sure if there's anything to automate the above processes.

User avatar
MasonFace
Posts: 346
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Thu Mar 28, 2019 3:51 am

Here is the result of your sprites composited with Gigapixel results:
pTsM6Qk_Gigapixel60_composite.png
pTsM6Qk_Gigapixel60_composite.png (88.66 KiB) Viewed 759 times
And below is the inn keeper with a quickly touched up face.
InnKeepTouchedUp.png
InnKeepTouchedUp.png (32.14 KiB) Viewed 759 times

phredreeke
Posts: 55
Joined: Mon Mar 11, 2019 9:13 am

Re: AI Upscaled Textures

Post by phredreeke » Thu Mar 28, 2019 10:34 am

MrFlibble wrote:
Wed Mar 27, 2019 9:06 pm
Honestly I've never given a thought about batching. When I was still using waifu2x my basic idea was to collect all sprites into a single sheet (keeping enough space between the sprites of course to prevent them from merging together) and then do all manipulations to that image. If there is a programme that stitches together multiple images (or a GIMP function that tiles layers) that would aid this kind of process immensely.
I use ImageMagick to combine upscales and masks

Code: Select all

md composite
for %%f in (*.png) do ("C:\Program Files\ImageMagick-7.0.7-Q8\Magick.exe" montage %%f "..\proc-downscalepalette2\%%f" -tile x1 -geometry +0+0 png32:composite\%%f)

MrFlibble
Posts: 129
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Thu Mar 28, 2019 11:58 am

MasonFace wrote:
Thu Mar 28, 2019 3:51 am
Here is the result of your sprites composited with Gigapixel results:
Interesting! Gigapixel seems to produce less cartoonish faces overall. I suspect it does have a semantic level somewhere inside which tries to identify parts of the image and then enhance them accordingly. But I cannot be certain since I've only seen some output from its processes and never did any testing of my own.

The girl looks a lot older though :) OTOH, very nice job on the hair.
MasonFace wrote:
Thu Mar 28, 2019 3:51 am
And below is the inn keeper with a quickly touched up face.
I think it's very cool!

Could you try the small NPC flats with Gigapixel too? I wonder how it will handle their faces.
phredreeke wrote:
Thu Mar 28, 2019 10:34 am
I use ImageMagick to combine upscales and masks

Code: Select all

md composite
for %%f in (*.png) do ("C:\Program Files\ImageMagick-7.0.7-Q8\Magick.exe" montage %%f "..\proc-downscalepalette2\%%f" -tile x1 -geometry +0+0 png32:composite\%%f)
Great, thanks for sharing this!!!

User avatar
MasonFace
Posts: 346
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Thu Mar 28, 2019 4:58 pm

I'm going to take a look at this new SC-FEGAN neural network this weekend. It looks really cool! Could help this project out with restoring the faces of the NPCs and maybe the portraits, too.

https://github.com/JoYoungjoo/SC-FEGAN

An example of what it can do with incomplete pictures of faces:
restoration.jpg
restoration.jpg (945.08 KiB) Viewed 704 times
Of course it was trained on actual photographs, so the results may be too realistic. Perhaps we can run some filters on the results to make it blend in a little better. Kind of a long shot, but could be interesting nonetheless.

User avatar
Alyndiar
Posts: 69
Joined: Thu Dec 13, 2018 1:22 am
Location: Québec

Re: AI Upscaled Textures

Post by Alyndiar » Fri Mar 29, 2019 9:51 pm

MasonFace wrote:
Thu Mar 28, 2019 4:58 pm
I'm going to take a look at this new SC-FEGAN neural network this weekend. It looks really cool! Could help this project out with restoring the faces of the NPCs and maybe the portraits, too.

https://github.com/JoYoungjoo/SC-FEGAN

An example of what it can do with incomplete pictures of faces:

restoration.jpg

Of course it was trained on actual photographs, so the results may be too realistic. Perhaps we can run some filters on the results to make it blend in a little better. Kind of a long shot, but could be interesting nonetheless.
Wow! If anyone can help me set this up on my side, I'd love to play with this! I just have no idea on how to make this work. But this is just amazing...

User avatar
MasonFace
Posts: 346
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Sat Mar 30, 2019 3:01 am

I'm not sure where to even begin to tell you how to get this running, but I can tell you how I got it working for me.

This is a scholarly project, so they tend to be focused more on results than making it easy for the uninitiated to install and run. You'll have to jump through several hoops to get it up and going.

I'm running Windows 10, so if you're using another operating system the instructions may be slightly different.

Important note: Artificial Intelligence algorithms are usually ran on the GPU because these "tensor type" systems are much more easily solved on GPU architecture. As of time of writing this, only NVIDIA graphics cards have the accompanying software to allow the AI to run on the GPU. This software is called CUDA and you will likely need to install this to speed up the calculations. If you don't have an NVIDIA graphics card, then you will have to run it on your CPU. Even the fastest CPUs are an order of magnitude slower than GPU for these calculations, so expect it to be slow. The authors don't specifically mention any of this, so it is possible that it somehow doesn't require CUDA to run on the GPU, but I believe it does.

The first thing you'll want to do is install Python 3: https://www.youtube.com/watch?v=3lGhtIqT0Tk

Now that you have Python installed, PIP should have been installed with it which you'll need to download all the other dependent packages.

Go to the SC-FEGAN Github page and look at the items in the Dependencies section:
https://github.com/JoYoungjoo/SC-FEGAN

Open command prompt on your computer. This is where we will enter our terminal commands. If you don't have a shortcut for command prompt saved, you can find it using the search bar next to your start menu at the bottom left corner of your desktop and searching for "CMD".

Inside of command prompt, use PIP to install each of the dependent packages that are on the dependencies list. For example, to install tensorflow, type "PIP install tensorflow" and press enter. Repeat this step for each item in the dependencies list.

Next, you will need to clone their Github repository. The easiest way is with Github desktop, but if you don't want to install that, you should be able to just download the repository by clicking on the green "Clone or downlaod" button near the top right of their Github page. Use the "Download ZIP" option. Extract the contents of the zip file wherever you like. Open the "SC_FEGAN" folder you just extracted and if it doesn't have a folder in there named ckpt, then create one. I don't know why they don't mention that in their instructions, but it almost threw me for a loop.

Next you will need to download their model. This contains all the trained weights that governs how the AI reacts to inputs. On their Github page, right beneath the Dependencies table, there is a link to the model that's hosted on their Google Drive account. Go there and download "SC-FEGAN.ckpt.data-00000-of-00001" into your SC_FEGAN/ckpt folder.

If you don't have and NVIDIA graphics card that supports CUDA, I believe you will have to run the program on your CPU instead. You will have to modify the file "demo.yaml" using notepad, change "GPU_NUM 1" to "GPU_NUM" (basically just erase the number). Save and close the file and you're good to go.

Now open up command prompt again. Navigate to your SC_FEGAN folder using the change directory command (cd) then type "python demo.py" and hit enter. It should run from there.
pythonrundemo.PNG
pythonrundemo.PNG (8.86 KiB) Viewed 626 times
If you have any trouble, just PM me so we don't clutter up the thread too much.

Post Reply