AI Upscaled Textures

Show off your mod creations or just a work in progress.
User avatar
MasonFace
Posts: 334
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Wed Aug 28, 2019 2:43 pm

Glad to see you back, MrFlibble!

Thanks for pointing to that collection of models. I've been in the dark on those too. It's really great to see people have been tirelessly dedicating their time to generate better and better models to share. I'm interested to check out that normal map generator model... I'm curious how it compares to traditional normal map algorithms.

I like the NPC upscales, and I especially like the fact that they do not require pre-processing!

NPCs might be the next category that I tackle and I will most likely use the Fatality/Rebout interpolation and touch-up the faces manually a little as I go.

Etherian
Posts: 2
Joined: Mon Aug 26, 2019 8:45 pm

Re: AI Upscaled Textures

Post by Etherian » Wed Aug 28, 2019 5:53 pm

MrFibble, that's excellent! I'd say those are the best pre-rendered flats yet. And among the handpainted, I think this setup is the first to really capture the feel of the old man with the cane.

Yeah, MasonFace, if you don't see a clear way for StyleGAN to benefit your existing pipeline, you're right – it's probably best not to bother with it.

I have one other suggestion, though, that might be more feasible. I noticed yall sometimes blend some of your methods together to merge their strengths and mitigate their weaknesses. Have you tried varying the ratio with which you blend them based on some metric in the image, like frequency or luminance? That way the result of a method could be emphasized where it works well, say faces, and minimized where it fails, say clothes.

User avatar
King of Worms
Posts: 2196
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms » Wed Aug 28, 2019 6:48 pm

Nice progress and whole AI clan back again :)
Thanks for the videos!

I think I will get Vegas and try some transitions of the static images... these vids (man, woman) and without sound?
If so, I will try to add some. Maybe from a existing Daggerfall soundtrack or something completely new..

User avatar
MasonFace
Posts: 334
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Wed Aug 28, 2019 6:50 pm

Have you tried varying the ratio with which you blend them based on some metric in the image, like frequency or luminance?
Unfortunately, where the various AI algorithms excel/fail is more feature based, such as "faces", "clothes", "hands", etc., so I don't think that's something we can key off of to drive blending ratios in specific zones of the image using traditional metrics. I think this would almost require an AI in itself to detect these features, then mask them off so that layer contributes less to the composited output.

All of that is out of my expertise, so I'm not going to invest any time on it. I haven't looked in a while, but I think since they aren't animated, the NPC flats are much smaller in number than other asset categories so manually fixing their faces may not take too much time. I'd still prefer an automated approach, but sometimes you just got to bite the bullet.

Keep thinking on it, though. You might think of something.

BTW, have you tried running ESRGAN? I think you might enjoy tinkering with it. I wrote an installation tutorial, but I'm sure there are many others out there by now that are much better.

Edit:

@KoW: I'm sure you can get better transition videos from the stills than I was able to do using OpenShot. I can only do the simplest things with it. You may want to recreate ANIM011 which is the text that fades in before the DAG2 live action video. I remade it, but I think it could be done a little better.

User avatar
MasonFace
Posts: 334
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Fri Aug 30, 2019 2:58 am

I'm really digging the results of the Fatality/Rebout models, especially interpolated together. Below is a random assortment of non cherry picked examples. Only a few of the several hundred portrait came out bad, so I'm pretty happy with that. :D
UpscaledBradyBunch.png
UpscaledBradyBunch.png (760.56 KiB) Viewed 862 times
It does extremely well on the NPC portraits and I think that is the key to quickly touching up the upscale NPC flats such as below.
Upscaled 47-0.png
Upscaled 47-0.png (401.41 KiB) Viewed 862 times
The results are quick and easy and look very vanilla friendly.

EDIT: I think it's important to point out that the test above is record 47-0 from archive 186 which is higher resolution than most of the other NPC flat archives, so this result is obviously going to be better than typical.

User avatar
King of Worms
Posts: 2196
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms » Fri Aug 30, 2019 8:32 pm

The portraits are crazy good for AI..
Damn!

MrFlibble
Posts: 114
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Sun Sep 01, 2019 1:43 pm

MasonFace wrote:
Fri Aug 30, 2019 2:58 am
It does extremely well on the NPC portraits and I think that is the key to quickly touching up the upscale NPC flats such as below.

Upscaled 47-0.png

The results are quick and easy and look very vanilla friendly.
I couldn't but notice those jiggety lines on the guy's right shoulder (to the viewer's left) and I previously had this suspicion that the black background does this, but ran a couple test and it turns out that you get smoother results with the additional interpolation with DeToon as I described above:
Image
Without DeToon I get the same results as you do:
Image
Also, the obligatory texture test, the +DeToon model on the right (same textures as used previously):
Image Image
Image Image
Image Image
Image Image
Image Image
I think that overall I tend to like the DeToon results more, although no idea if the textures would actually look better in-game.

The +DeToon model was produced by interpolating the Fatality/Rebout_interp interpolated model with detoon_alt, again at 0.5.

User avatar
MasonFace
Posts: 334
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Sun Sep 01, 2019 5:58 pm

Yeah, I think you're right that the DeToon helps to soften the result better. How are you compounding the interpolations? Are you running net_interp.py 0.5 with Fatality and Rebout, then running net_interp.py 0.5 with interp_05.pth and Detoon?

MrFlibble
Posts: 114
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Tue Sep 03, 2019 4:57 pm

MasonFace wrote:
Sun Sep 01, 2019 5:58 pm
Yeah, I think you're right that the DeToon helps to soften the result better. How are you compounding the interpolations? Are you running net_interp.py 0.5 with Fatality and Rebout, then running net_interp.py 0.5 with interp_05.pth and Detoon?
Basically yes, except of course I renamed interp_05.pth to something else (I used F_RB_05.pth for easy typing).

Here's the net_interp.py code I used:

Code: Select all

net_PSNR_path = './models/rebout_interp.pth'
net_ESRGAN_path = './models/4x_Fatality_01_265000_G.pth'

Code: Select all

net_PSNR_path = './models/detoon_alt.pth'
net_ESRGAN_path = './models/F_RB_05.pth'
I think it does make a difference which model you put as 'ESRGAN' and which as 'PSNR'; basically if I say interpolate X with Y, I mean that X is in the ESRGAN path and Y in the PSNR path.

Note though that after some further testing I came to realise that DeToon is not always better than simple Fatality/Rebout, as it introduces a kind of noise that may distort image detail. For example this test with Wolfenstein 3-D title screen shows no advantage of the DeToon effect while the guard's face came out less detailed and fuzzy.

User avatar
King of Worms
Posts: 2196
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms » Sun Sep 08, 2019 9:21 am

phredreeke wrote:
Mon Aug 26, 2019 11:13 pm
alright, did renders of 03,05,12 and 13 with the new script

https://www.dropbox.com/sh/1056i09ys0cx ... 4Ty0a?dl=0

(please download ASAP as I may have to delete it to make space)

Something to consider 1) their final resolution is 1440x1080 but most of the script runs at a lower resolution (1280 wide, height depending on the size of the picture area) since I found that to give better result when uploading to youtube but since the video runs in-game it shouldn't matter much, so what would the desired target resolution be? 2) while the latest script crops the top and bottom to reduce processing times it adds them back near the end to get a 4:3 ratio, should it instead be kept at whatever ratio the image area is?

for the book videos, I think the best course of action would be to try to remake them. the illustrations turns out alright IMO but the text looks pretty ugly (the standalone frames I uploaded are using the fatality model)

a final thing to consider, maybe for certain static shots (the start of anim0012 for example) touch up parts and drop them into the upscale (like the tomb/temple on the right side of the screen)

I'll provide frames for the raven tomorrow.
Hi, the vids are really nice thank you (and the stills as well)
But they play in smaller part of the screen than the originals ingame. I think its because of the 1440x1080 aspect ratio.

From my experience the 1920x1080 and 1280x720 play properly sized.

I was thinking - instead of me resizing them and making em more blurry, it would be cool if you could re-render them.
But if thats something you dont feel like doing, I completely understand that.

Also, I think the videos with books and turning pages (the one from which only the still frames you uploaded) would be just as good when upscaled the way you did the rest. I dont see a way to use those stills honestly.

Post Reply