Page 1 of 54

AI Upscaled Textures

Posted: Fri Dec 21, 2018 10:06 pm
by MasonFace
UPDATE:

I'm going to try to keep this first post updated with the latest on this project.

---------------------------------------------------------------------

UPDATE:

Satsuki has created a great GUI for ESRGAN. It can even handle pre-filtering! ESRGAN Resizer

Summary:

We are trying to upscale the original Daggerfall textures using a variety of artificial intelligence techniques and some combination of image filters.

All textures have already been upscaled by Freak2121 using Topaz A.I. Gigapixel.
You can download that here: Freak2121's Texture Pack (Gigapixel 4x)

MrFlibble and I have been using various models and pre-processing techniques to enhance upscales using ESRGAN and SFTGAN.

I have created further enhanced textures for Privateer's Hold and created advanced materials for them. You can download this texture replacement mini-mod here (screenshot below):
Masonface's Upscaled Texture Pack (4x + PBR materials) - Privateer's Hold

Note: Make sure filtering mode is set to bilinear or trilinear, not point!

Screenshot:
Screen4.jpg
Screen4.jpg (271.81 KiB) Viewed 3263 times
---------------------------------------------------------------------

Methodology:

As of now, Freak2121 is getting great results out-of-the-box on textures using Topaz's A.I. Gigapixel.

Other techniques that we've employed are ESRGAN and SFTGAN with mixed results depending on the source image, the model used, and any pre or post processing filters used. These techniques could prove to be the best route if we can train the model on more appropriate data so that it produces results that are higher definition, yet retain the correct aesthetic.

My opinion is that all generated textures/sprites that have been contributed should be made public to the community so anyone can curate their own collection.

The idea there-after is to compile an official collection of textures from all the fore-mentioned techniques, then curate a singular collection that is agreed to best fit for convenience to end-users.

---------------------------------------------------------------------

Original Post:

By suggestion of Luzur, I started tinkering around with some experimental artificial intelligence upscaling methods.

I just started with some basic tests using ESRGAN (https://github.com/xinntao/ESRGAN), and while they aren't perfect, they really ain't half bad! Take a look:

https://imgur.com/zHqCYr0

https://imgur.com/QD83bTw

I think this would be a good way to get a high quality pack of textures rolled out quickly while talented artists take their time remastering the originals (which I think most of us would prefer).

Next is a test using SFTGAN (https://github.com/xinntao/SFTGAN) which I think looks a bit better for textures:

https://imgur.com/YsyXfEe

Re: AI Upscaled Textures

Posted: Fri Dec 21, 2018 10:44 pm
by Interkarma
Thanks for looking into that and sharing results on DF textures. Very fascinating stuff.

I'm amazed how ESRGAN adds those scratches and chips to the left and right sides of window arch, despite those features only being hinted at in original texture. It does take away some of the hand-painted feel in the pixel textures though. But you're right, it's not half bad either.

The SFTGAN does feel a lot more like the original in terms of hand-painted feel to me.

Both of them have trouble with the roughly sculpted arch in original image, but that's more an issue with fidelity of original than anything. Considering the original arched window texture is only 64x128 pixels, very cool!

Re: AI Upscaled Textures

Posted: Fri Dec 21, 2018 11:06 pm
by Hazelnut
Is using these tools at runtime feasible do you think?

Re: AI Upscaled Textures

Posted: Sat Dec 22, 2018 4:19 am
by Wysardry
I'm far from an expert on graphics, but couldn't human artists tweak the images produced by the AI program instead of starting with the originals? I would imagine it would save them quite a lot of time and effort, yet still give a more handmade look.

Re: AI Upscaled Textures

Posted: Sat Dec 22, 2018 7:40 am
by VMblast
As an artist I can only say that this is wonderful! :)

Someone would say to that -what?! But, for a long answer -while upscale like these to someone could look strange, somewhat skewed and far from realistic (painted) look, to me these are wonderful works of impressionism! Now, the classic (human) painting can never be replaced as only artist can do precise work of something looking like beam, window, floor tile, grass etc. But this is really good and interesting stylized alternative. I would very much like to see all the textures redone by these. :)

PS - While both systems are great, Im leaning more to SFTGAN version, as it gives better (oil) painted look (you can almost see the brush strokes). Dont get me wrong, ESRGAN is awesome as well, as it is sharper and more illustrative (sketchy -maybe just a tad too contrasty). It is hard to choose between these two, so I would say mixed application would be the best way.

Im looking forward to this experiment as a MOD!

Re: AI Upscaled Textures

Posted: Sat Dec 22, 2018 7:59 am
by pango
It may matter to mention not only the algorithms used, but also the models used, as the results depend on the models too (indirectly, on the examples that were learned to produce the model).

Have you done some tests on NPCs/monsters too?

Re: AI Upscaled Textures

Posted: Sat Dec 22, 2018 11:21 am
by King of Worms
Looks interesting, the later example looks better to me.
Is there any download which includes the UI?
Id like to experiment with it, but I dont have any idea how to use the files from the git you linked.
Thank you

Re: AI Upscaled Textures

Posted: Sun Dec 23, 2018 2:05 am
by MrDowntempo
there is a similar experiment going on with doom sprites. However, they're using proprietary tools instead of open source ones. They're also doing a lot of work after the fact, specifically, layered one algorithm's result on top of another. They're also upscaling a LOT and then manually scaling it back down. I think they're upscaling like 8x and then scaling it down so that the final result is 2x larger than the original. It sort of filters out t he weird AI artifacts that way. Might be worth trying some of those strategies.

Re: AI Upscaled Textures

Posted: Sun Dec 23, 2018 2:55 am
by MasonFace
Is using these tools at runtime feasible do you think?
Unfortunately, no. Running the upscaling algorithm doesn't take very long to generate each result, but it requires having CUDA installed and having a relatively modern Nvidia graphics card. Also, I think it would take too long to generate ALL the textures needed at run-time.
I'm far from an expert on graphics, but couldn't human artists tweak the images produced by the AI program instead of starting with the originals? I would imagine it would save them quite a lot of time and effort, yet still give a more handmade look.
Yes, it could save some time depending on the particular artist's workflow.
As an artist I can only say that this is wonderful! :)

Someone would say to that -what?! But, for a long answer -while upscale like these to someone could look strange, somewhat skewed and far from realistic (painted) look, to me these are wonderful works of impressionism! Now, the classic (human) painting can never be replaced as only artist can do precise work of something looking like beam, window, floor tile, grass etc. But this is really good and interesting stylized alternative. I would very much like to see all the textures redone by these. :)

PS - While both systems are great, Im leaning more to SFTGAN version, as it gives better (oil) painted look (you can almost see the brush strokes). Dont get me wrong, ESRGAN is awesome as well, as it is sharper and more illustrative (sketchy -maybe just a tad too contrasty). It is hard to choose between these two, so I would say mixed application would be the best way.
I think based on the way these two models were trained, they will have differing pros and cons depending on the input textures. For example, it looks like ESRGAN is interpreting the dark regions of the marble to be cracks, where the SFTGAN seems to have a better grasp of what marble is and looks like while also adding in more subtle surface wear to the columns. On stone and earth textures, ESRGAN may fair better. I'll perform a larger sample set and see how they compare in various types of textures, but the SFTGAN workflow is much more tedious and takes longer to get the input textures ready. For ESRGAN, I just have to put the original textures in a folder, then run a single python script and it handles the upscaling. For SFTGAN, I have to first scale the image up to its final size, then run a python script, then run another python script before it begins the upscaling.
It may matter to mention not only the algorithms used, but also the models used, as the results depend on the models too (indirectly, on the examples that were learned to produce the model).
From what I read, the SFTGAN that was used in the referenced Git was trained with lots of outdoor photos and that's where it excels. I think that makes it a better fit for most Daggerfall textures overall, but it may not always be the case. Of course, if we had a LARGE collection of textures that have already been hand upscaled by artists, we could train the AI to be better at upscaling the remaining low resolution Daggerfall textures... but I think we would need a LOT of upscaled samples (like maybe 1,000 or more).
Have you done some tests on NPCs/monsters too?
I only tried one NPC using the ESRGAN, and the result was really bad.
Looks interesting, the later example looks better to me.
Is there any download which includes the UI?
Id like to experiment with it, but I dont have any idea how to use the files from the git you linked.
Yeah, I like the SFTGAN better too on these examples.

There isn't a UI that makes any of this easy; I had to jump through a lot of hoops to get it to run. You need to be fairly comfortable with using terminal commands and you need to have a modern Nvidia graphics card, otherwise you are limited to running the AI (tensor type calculations) on the CPU which is extremely slow.

If you want to give it a go, follow the instructions here: https://www.resetera.com/threads/ai-neu ... elf.88272/

Let me know if you need any help and I'll assist you as best I can (at least if you're using Windows 10).
there is a similar experiment going on with doom sprites. However, they're using proprietary tools instead of open source ones. They're also doing a lot of work after the fact, specifically, layered one algorithm's result on top of another. They're also upscaling a LOT and then manually scaling it back down. I think they're upscaling like 8x and then scaling it down so that the final result is 2x larger than the original. It sort of filters out t he weird AI artifacts that way. Might be worth trying some of those strategies.
Yeah, I think they're using the official Nvidia tool which is in closed beta. It would be nice if we had access to it... :cry:
I had considered layering the two upscaled results on top of each other and maybe using a soft erase brush on the worse looking layer to let the better result show through at that region, but I hadn't considered upscaling to 8x then back down. That's an interesting technique. Thanks for bringing that up!

Re: AI Upscaled Textures

Posted: Sun Dec 23, 2018 12:13 pm
by King of Worms
MrDowntempo wrote:
Sun Dec 23, 2018 2:05 am
Might be worth trying some of those strategies.
In my remaster, I utilized the BRZ4x in two passes, so that means 16x upscale and than manual downscale... so thats a similar aproach. Interresting idea about stacking two algo results in one sprite btw

BTW I applyed for that closed beta nvidia program, and they did not like me I guess, no reply in months

This AI upscaling will be a future of DFU one day, it can produce INSANE results... I saw those at the link u posted, damn!
46221-1544831305-62299445.png
46221-1544831305-62299445.png (181.89 KiB) Viewed 5035 times
46221-1544880920-600360147.png
46221-1544880920-600360147.png (595.78 KiB) Viewed 5035 times