AI Upscaled Textures
- MasonFace
- Posts: 543
- Joined: Tue Nov 27, 2018 7:28 pm
- Location: Tennessee, USA
- Contact:
Re: AI Upscaled Textures
Glad to see you back, MrFlibble!
Thanks for pointing to that collection of models. I've been in the dark on those too. It's really great to see people have been tirelessly dedicating their time to generate better and better models to share. I'm interested to check out that normal map generator model... I'm curious how it compares to traditional normal map algorithms.
I like the NPC upscales, and I especially like the fact that they do not require pre-processing!
NPCs might be the next category that I tackle and I will most likely use the Fatality/Rebout interpolation and touch-up the faces manually a little as I go.
Thanks for pointing to that collection of models. I've been in the dark on those too. It's really great to see people have been tirelessly dedicating their time to generate better and better models to share. I'm interested to check out that normal map generator model... I'm curious how it compares to traditional normal map algorithms.
I like the NPC upscales, and I especially like the fact that they do not require pre-processing!
NPCs might be the next category that I tackle and I will most likely use the Fatality/Rebout interpolation and touch-up the faces manually a little as I go.
-
- Posts: 2
- Joined: Mon Aug 26, 2019 8:45 pm
Re: AI Upscaled Textures
MrFibble, that's excellent! I'd say those are the best pre-rendered flats yet. And among the handpainted, I think this setup is the first to really capture the feel of the old man with the cane.
Yeah, MasonFace, if you don't see a clear way for StyleGAN to benefit your existing pipeline, you're right – it's probably best not to bother with it.
I have one other suggestion, though, that might be more feasible. I noticed yall sometimes blend some of your methods together to merge their strengths and mitigate their weaknesses. Have you tried varying the ratio with which you blend them based on some metric in the image, like frequency or luminance? That way the result of a method could be emphasized where it works well, say faces, and minimized where it fails, say clothes.
Yeah, MasonFace, if you don't see a clear way for StyleGAN to benefit your existing pipeline, you're right – it's probably best not to bother with it.
I have one other suggestion, though, that might be more feasible. I noticed yall sometimes blend some of your methods together to merge their strengths and mitigate their weaknesses. Have you tried varying the ratio with which you blend them based on some metric in the image, like frequency or luminance? That way the result of a method could be emphasized where it works well, say faces, and minimized where it fails, say clothes.
- King of Worms
- Posts: 4753
- Joined: Mon Oct 17, 2016 11:18 pm
- Location: Scourg Barrow (CZ)
- Contact:
Re: AI Upscaled Textures
Nice progress and whole AI clan back again
Thanks for the videos!
I think I will get Vegas and try some transitions of the static images... these vids (man, woman) and without sound?
If so, I will try to add some. Maybe from a existing Daggerfall soundtrack or something completely new..
Thanks for the videos!
I think I will get Vegas and try some transitions of the static images... these vids (man, woman) and without sound?
If so, I will try to add some. Maybe from a existing Daggerfall soundtrack or something completely new..
DREAM Nexus https://www.nexusmods.com/users/73250038?tab=user+files
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
- MasonFace
- Posts: 543
- Joined: Tue Nov 27, 2018 7:28 pm
- Location: Tennessee, USA
- Contact:
Re: AI Upscaled Textures
Unfortunately, where the various AI algorithms excel/fail is more feature based, such as "faces", "clothes", "hands", etc., so I don't think that's something we can key off of to drive blending ratios in specific zones of the image using traditional metrics. I think this would almost require an AI in itself to detect these features, then mask them off so that layer contributes less to the composited output.Have you tried varying the ratio with which you blend them based on some metric in the image, like frequency or luminance?
All of that is out of my expertise, so I'm not going to invest any time on it. I haven't looked in a while, but I think since they aren't animated, the NPC flats are much smaller in number than other asset categories so manually fixing their faces may not take too much time. I'd still prefer an automated approach, but sometimes you just got to bite the bullet.
Keep thinking on it, though. You might think of something.
BTW, have you tried running ESRGAN? I think you might enjoy tinkering with it. I wrote an installation tutorial, but I'm sure there are many others out there by now that are much better.
Edit:
@KoW: I'm sure you can get better transition videos from the stills than I was able to do using OpenShot. I can only do the simplest things with it. You may want to recreate ANIM011 which is the text that fades in before the DAG2 live action video. I remade it, but I think it could be done a little better.
- MasonFace
- Posts: 543
- Joined: Tue Nov 27, 2018 7:28 pm
- Location: Tennessee, USA
- Contact:
Re: AI Upscaled Textures
I'm really digging the results of the Fatality/Rebout models, especially interpolated together. Below is a random assortment of non cherry picked examples. Only a few of the several hundred portrait came out bad, so I'm pretty happy with that.
It does extremely well on the NPC portraits and I think that is the key to quickly touching up the upscale NPC flats such as below.
The results are quick and easy and look very vanilla friendly.
EDIT: I think it's important to point out that the test above is record 47-0 from archive 186 which is higher resolution than most of the other NPC flat archives, so this result is obviously going to be better than typical.
It does extremely well on the NPC portraits and I think that is the key to quickly touching up the upscale NPC flats such as below.
The results are quick and easy and look very vanilla friendly.
EDIT: I think it's important to point out that the test above is record 47-0 from archive 186 which is higher resolution than most of the other NPC flat archives, so this result is obviously going to be better than typical.
- King of Worms
- Posts: 4753
- Joined: Mon Oct 17, 2016 11:18 pm
- Location: Scourg Barrow (CZ)
- Contact:
Re: AI Upscaled Textures
The portraits are crazy good for AI..
Damn!
Damn!
DREAM Nexus https://www.nexusmods.com/users/73250038?tab=user+files
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
- MrFlibble
- Posts: 411
- Joined: Sat Jan 27, 2018 10:43 am
Re: AI Upscaled Textures
I couldn't but notice those jiggety lines on the guy's right shoulder (to the viewer's left) and I previously had this suspicion that the black background does this, but ran a couple test and it turns out that you get smoother results with the additional interpolation with DeToon as I described above:
Without DeToon I get the same results as you do:
Also, the obligatory texture test, the +DeToon model on the right (same textures as used previously):
I think that overall I tend to like the DeToon results more, although no idea if the textures would actually look better in-game.
The +DeToon model was produced by interpolating the Fatality/Rebout_interp interpolated model with detoon_alt, again at 0.5.
- MasonFace
- Posts: 543
- Joined: Tue Nov 27, 2018 7:28 pm
- Location: Tennessee, USA
- Contact:
Re: AI Upscaled Textures
Yeah, I think you're right that the DeToon helps to soften the result better. How are you compounding the interpolations? Are you running net_interp.py 0.5 with Fatality and Rebout, then running net_interp.py 0.5 with interp_05.pth and Detoon?
- MrFlibble
- Posts: 411
- Joined: Sat Jan 27, 2018 10:43 am
Re: AI Upscaled Textures
Basically yes, except of course I renamed interp_05.pth to something else (I used F_RB_05.pth for easy typing).
Here's the net_interp.py code I used:
Code: Select all
net_PSNR_path = './models/rebout_interp.pth'
net_ESRGAN_path = './models/4x_Fatality_01_265000_G.pth'
Code: Select all
net_PSNR_path = './models/detoon_alt.pth'
net_ESRGAN_path = './models/F_RB_05.pth'
Note though that after some further testing I came to realise that DeToon is not always better than simple Fatality/Rebout, as it introduces a kind of noise that may distort image detail. For example this test with Wolfenstein 3-D title screen shows no advantage of the DeToon effect while the guard's face came out less detailed and fuzzy.
- King of Worms
- Posts: 4753
- Joined: Mon Oct 17, 2016 11:18 pm
- Location: Scourg Barrow (CZ)
- Contact:
Re: AI Upscaled Textures
Hi, the vids are really nice thank you (and the stills as well)phredreeke wrote: ↑Mon Aug 26, 2019 11:13 pm alright, did renders of 03,05,12 and 13 with the new script
https://www.dropbox.com/sh/1056i09ys0cx ... 4Ty0a?dl=0
(please download ASAP as I may have to delete it to make space)
Something to consider 1) their final resolution is 1440x1080 but most of the script runs at a lower resolution (1280 wide, height depending on the size of the picture area) since I found that to give better result when uploading to youtube but since the video runs in-game it shouldn't matter much, so what would the desired target resolution be? 2) while the latest script crops the top and bottom to reduce processing times it adds them back near the end to get a 4:3 ratio, should it instead be kept at whatever ratio the image area is?
for the book videos, I think the best course of action would be to try to remake them. the illustrations turns out alright IMO but the text looks pretty ugly (the standalone frames I uploaded are using the fatality model)
a final thing to consider, maybe for certain static shots (the start of anim0012 for example) touch up parts and drop them into the upscale (like the tomb/temple on the right side of the screen)
I'll provide frames for the raven tomorrow.
But they play in smaller part of the screen than the originals ingame. I think its because of the 1440x1080 aspect ratio.
From my experience the 1920x1080 and 1280x720 play properly sized.
I was thinking - instead of me resizing them and making em more blurry, it would be cool if you could re-render them.
But if thats something you dont feel like doing, I completely understand that.
Also, I think the videos with books and turning pages (the one from which only the still frames you uploaded) would be just as good when upscaled the way you did the rest. I dont see a way to use those stills honestly.
DREAM Nexus https://www.nexusmods.com/users/73250038?tab=user+files
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz