AI Upscaled Textures

Show off your mod creations or just a work in progress.
Post Reply
User avatar
King of Worms
Posts: 4752
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

I would add that the textures need to be able to be divided by 4 on both dimensions. Its best to stick to standards like VMBlast mentioned
128
256
512
1024
2048

It saves a lot of time afterwards...

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: AI Upscaled Textures

Post by MasonFace »

One thing tho, textures need to be at least 512x1024 to work properly ingame. What I mean is that for the monitor HD resolution of 1920x1080 is best visually supported by (minimal) vertical texture size of 1024.
I would add that the textures need to be able to be divided by 4 on both dimensions. Its best to stick to standards like VMBlast mentioned
Thanks for reminding me! My upscales were hitting weird resolutions and I couldn't figure out why they weren't looking quite right in-game. They look better now. BTW, any idea when custom streaming world texture injection will be supported?
Id like to see some enemy sprites, or npcs, or tree sprites...
I've been poking around with this and it the results are very hit-or-miss (mostly miss) when it comes to sprites. The best I've gotten so far is the horse, and it needs a bit of manual cleaning up around the silhouette and it would look pretty good.

Image
I first used blur -> ESRGAN and the results were pretty good.

Image
Then I used just SFTGAN and got much better detail, but a little worse silhouette.

The second best (and it's a far second) was the dog. The problem seems to be that there just isn't nearly enough pixels to start off with (a measly 56x40!) for the AI to infer any additional detail from...

Image
It doesn't look any better than just a regular cubic interpolated scaling...

I tried a few trees and bushes, but they all came out either just terribly blurry or highly contaminated with artifacts.

I may try a few more experiments on the sprites to see if I can get consistently decent results. Otherwise, I'll continue to focus on textures.
Hmm... Train it on manually-upscaled textures, then feed in the original textures?
Yup, that's the end-game! Would you be able to help? I need a compilation of textures close to or above our target resolution (1024x1024 would probably be okay), preferably of hand-painted textures of a variety of materials (wood, stone, brick, marble, metal, etc.). I will need a bare minimum of 100, but we will get increasingly better results with more training data. I will continue to read up on how to actually perform the training... from what I understand, that is the actual computationally expensive part of the AI.

Al-Khwarizmi
Posts: 177
Joined: Sun Mar 22, 2015 9:52 am

Re: AI Upscaled Textures

Post by Al-Khwarizmi »

Wow, how does SFTGAN know the horse's eye should actually be an eye, with round pupil and gleam and all, from only six pixels? I use neural networks at my job but I never stop being amazed.

User avatar
King of Worms
Posts: 4752
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

SFTGAN horse is quite ok.

Well... I have a idea... I can give you some upscaled versions of the original art so you can train the AI on it.

Im sure my NPCs and Sprites are the best learning material for it. Its VERY vanilla and high quality compared to source materials.

Than I have upscaled textures...

Im not so sure with them in terms of AI learning. Some of them are vanila upscaled with XBR4x... some of them are redrawn.

Tell me where to start.

NPC Example
Spoiler!
001.jpg
001.jpg (385.01 KiB) Viewed 7069 times

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: AI Upscaled Textures

Post by MasonFace »

.


TL;DR - Let's start with textures. I'll need at least 100 unique hand-painted textures.

I think we would do best to train separate models: one for textures, another for NPCs/animals, and maybe another for vegetation/foliage. And despite what we've seen so far, I actually think ESRGAN would win out over SFTGAN if we train the models specific to our tasks, but I'd like to train both ESRGAN and SFTGAN with the same training data and see how they compare. I think the best place to start would be textures - seems it would be the least challenging to find enough training data to get useful results (100-500 unique samples I'm thinking).

First, we need to define what outcome we are shooting for; I personally prefer the "hand-painted" look for Daggerfall textures over the more realistic ones. We can import collections of hand-painted textures that all look approximately the same style, including those you and VMBlast have created. This set would be our "ground truth" samples and will be the target that the AI is striving to achieve. The training inputs will simply be the ground truth samples downscaled to vanilla DF resolution. It would be helpful if the hand-painted ground truth samples contain various materials (such as metal, marble, wood, stone, grass, etc.) and various features (pillars, beams, bolts/nail heads, window frames, cracks, etc.). Over the course of the training process, the GAN will "learn" how to fill in all the missing information during the upscale.

Again, the hand-painted ground truth samples don't have to be just previously upscaled DF textures. If you guys have any prior art or know of a royalty-free source of these types of textures, they will probably work just fine.

In the meantime, I'm going to mess around with some other upscaling techniques such as Waifu2x and XBR4x like you mentioned and see if I can somehow combine them with these GANs to enhance the output a little.

User avatar
VMblast
Posts: 519
Joined: Wed Mar 29, 2017 12:22 pm
Contact:

Re: AI Upscaled Textures

Post by VMblast »

MasonFace wrote: Wed Dec 26, 2018 4:32 am The second best (and it's a far second) was the dog. The problem seems to be that there just isn't nearly enough pixels to start off with (a measly 56x40!) for the AI to infer any additional detail from...

Image
It doesn't look any better than just a regular cubic interpolated scaling...

I tried a few trees and bushes, but they all came out either just terribly blurry or highly contaminated with artifacts.
What I meant when I mentioned HD resolution with upscaling to 1024, should be taken with a grain of salt. The resolution of 1024 vertical was aimed at textures which are the biggest in the ingame world, in the case of examples, these would be walls of the tavern interior or exterior. Anything (vanilla textures) smaller than that percentage vise, should be upscaled accordingly.

Vanilla: 64x128 should be up to min. ---> 512x1024

If the dog texture was 64x64, percentage wise, upscale should be 512x512 (since it covers smaler portion of the actual HD screen, when compared to the walls textures). ;)

PS -All the upscaling are looking good so far. A bit bury sometimes, but you are on the right track. Shame that no one havent made actual simple app for easier use.

Narf the Mouse
Posts: 833
Joined: Mon Nov 30, 2015 6:32 pm

Re: AI Upscaled Textures

Post by Narf the Mouse »

MasonFace wrote: Wed Dec 26, 2018 4:32 am
Hmm... Train it on manually-upscaled textures, then feed in the original textures?
Yup, that's the end-game! Would you be able to help? I need a compilation of textures close to or above our target resolution (1024x1024 would probably be okay), preferably of hand-painted textures of a variety of materials (wood, stone, brick, marble, metal, etc.). I will need a bare minimum of 100, but we will get increasingly better results with more training data. I will continue to read up on how to actually perform the training... from what I understand, that is the actual computationally expensive part of the AI.
Unfortunately outside my area of expertise; I've just read some summary articles and watched a few cool Youtube videos. That opinion is about the farthest I can venture right now. ;)

...Oh, and I also once trained a 5-neuron? NAND gate from a tutorial about 5? 6? years ago. Heh. Anyway, I'd unfortunately have to spend quite a bit more time than I have training myself up on the techniques and technologies. :)

But if you've got a training set you want run, and all I have to do is push a few buttons, I have a Ryzen 2700X. :)
Previous experience tells me it's very easy to misunderstand the tone, intent, or meaning of what I've posted. If you have questions, ask.

User avatar
Freak2121
Posts: 60
Joined: Fri Sep 08, 2017 6:14 am

Re: AI Upscaled Textures

Post by Freak2121 »

This stuff is cool! I've always wondered how well AI upscaling would work for old games like Daggerfall, nice to see it works quite well. I imagine if it gets trained on actual sprites and textures rather than real photos, the results would be incredible.

Anyway, I was curious so I signed up for a 30 day trial with Topaz AI's upscaler, and the results are quite nice. Here's a bit of a dump of some textures and sprites I tested it on, it seems to do really well with flora and really bad with faces. Mostly Daggerfall, though a few are from other games.

Warning, a whole ton of images in the spoiler.
Spoiler!
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Note the bottle on the bottom, second from left. It just did an incredible job on that one in particular
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Honestly these would all be really good with a bit of manual retouching.

User avatar
Arctus
Posts: 60
Joined: Mon Mar 23, 2015 12:28 am

Re: AI Upscaled Textures

Post by Arctus »

Damn those look great for procedural upscaling. So the entirety of DF textures and sprites could be run through that?

User avatar
Interkarma
Posts: 7236
Joined: Sun Mar 22, 2015 1:51 am

Re: AI Upscaled Textures

Post by Interkarma »

Wow, these look incredible for an AI upscaler. Some of those faces are a bit unsettling, but nothing a manual touch-up can't fix later. Excellent work!

Post Reply