AI Upscaled Textures

Show off your mod creations or just a work in progress.
Post Reply
phredreeke
Posts: 71
Joined: Mon Mar 11, 2019 9:13 am

Re: AI Upscaled Textures

Post by phredreeke »

I decided to give the previous faces another shot - this time with an interpolation of Spongebob Reloaded Swag (as used on the magazine in the upscale on last page) and Lollypop models. Afterwards I used imagemagick to remap the result to the palette of the original image.
face-compare.png
face-compare.png (15.49 KiB) Viewed 3537 times
Attachments
lollypop-spongebobswag-palette.zip
(258.69 KiB) Downloaded 131 times

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

Had a look thru them and they are nice, I know how difficult are these faces for AI so nice job.

I will personally not use them, because my goal for my mod is to have them all manually redrawn in HD (and that goal is lets say 80% ready)

Still, really impressive results indeed.

I wonder what results we could get on MOBs with these new algos, but I dont even dare to go there anymore :D I still think what we have is really good, especially the front facing frames which had a Topaz AI sharpening pass and lots of manual input after the AI upscale.
Spoiler!
260_0-0.png
260_0-0.png (401.93 KiB) Viewed 3509 times
265_0-0.png
265_0-0.png (287.13 KiB) Viewed 3509 times
277_0-3.png
277_0-3.png (810.06 KiB) Viewed 3509 times

phredreeke
Posts: 71
Joined: Mon Mar 11, 2019 9:13 am

Re: AI Upscaled Textures

Post by phredreeke »

Thought I'd share these. The results of an experimental model that been shared with me.
df-face1.png
df-face1.png (95.11 KiB) Viewed 3371 times
df-face2.png
df-face2.png (83.76 KiB) Viewed 3371 times

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

This looks awesome!

I think with a few touches of a artist, these are really good. I wonder how hard would be to remove the black background (some algos are hard to automatically cut-out)

I miss last 23 faces. Can I please send em to you for this treatment?
Even if I need to cut em out manually afterwards, its doable.

In case you could do it, I will add a RAR with these faces :)
Thanks a lot! (& even if just some of those come out good, its still few new faces done, I really need to push this forward somehow)

Here goes:
Faces to do.rar
(24.02 KiB) Downloaded 147 times

phredreeke
Posts: 71
Joined: Mon Mar 11, 2019 9:13 am

Re: AI Upscaled Textures

Post by phredreeke »

sure, here are the upscales
Attachments
tinyFace_Big_140k-2.zip
(1.15 MiB) Downloaded 135 times
tinyFace_Big_140k.zip
(980.91 KiB) Downloaded 127 times

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

Thanks a lot sir, will go thru them when I have some more free time, really appreciated 😊👍

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

I'm awaiting on the release of a newer model from Nvidia that should be open sourced in the next couple of weeks for turning these results into 3D renders--but thought I'd share in the meantime. Humans only unfortunately--will need to work through the process of style transfer for non human NPCs (retraining).

I used pixel2style2pixel to generate the photorealistic human faces from an older download of the NPC upscaled sprites portraits. The one on the right is a render of the transferred style--I'm pretty sure there is a way to transfer an art style too, so if someone wanted to apply a non lore-breaking "flavor" of art to these, it could be done to make the originals look whatever style you want.

hopefully by the time the 3Dganiverse model is out I can have the remaining fine tuning for faces done and apply another StyleGan model to the NPC flats, fully automate the process of converting them to 3D models.

Edit: the point of this is to demonstrate for faces, that there is a way to generate a template human photo of a face resembling the original to then apply an additional art style that fits the original game art using something like StyleGAN2. Many of the networks take photos of humans as input--this is a way to generate those photos for use in a larger pipeline. The option to face swap onto the full body sprites after applying a StyleGAN2 layer for an artistic style is easy to do--I'll try to do an example over the weekend, time permitting.
Screenshot_2021-06-18_22-43-19.png
Screenshot_2021-06-18_22-43-19.png (768.11 KiB) Viewed 2246 times

User avatar
Merlkir
Posts: 25
Joined: Thu Dec 28, 2017 12:23 pm
Contact:

Re: AI Upscaled Textures

Post by Merlkir »

Oh dang, those are surprisingly decent results.
Now I'm wondering what these would look like with the additional style step.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Yeah I am thinking the approach of super resolution needs to happen *in conjunction* with style transfer to attain the best results. I may be wrong but it's fun to play with in my free time anyway lol.

For the non humans I am considering extracting encodings of styles from images of Modern TES games, applying that style to the remaining non human NPCs. Then applying an extracted high res DF style (or a combination of styles) to all.

Additionally:
I'm thinking it might be fun to to achieve a good balance between the style of the game with the high resolution of form from actual photos + modern TES extractions. If the modern TES extractions may be foggy on licensing for using them here, I can automatically generate variations of a single extremely high resolution 3D model that I can put together myself (for anyone to use for commercial or non under MIT license), then encode those (custom dataset, would add a couple hours to generate over just using TES extractions from web scraped images).

Style and "Form" here (there's another word for "form" but I can't recall it off the top of my head) refers to 2 things, style being the art style and "Form" being sort of like the type of physical object that you are transforming the original sprite into (or applying a style to).

My understanding of StyleGAN2 is very high level at the moment, I'm just reading about models, playing with and retraining them where I can find usable code, and plugging them together using Python/OpenCV/PIL. I've trained YOLOv3 and YOLOv4 to detect body parts and typically use that for automatically cropping things prior to applying something this--made a repo tutorial for training YOLO here (works for V4 too):

https://github.com/WyattAutomation/Trai ... enImagesV4

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: AI Upscaled Textures

Post by MasonFace »

Really impressive results.

Honestly, just simply overlaying the GAN faces over the originals looks pretty dang good.
Composite.png
Composite.png (276.83 KiB) Viewed 2098 times
But I'm excited to see what the GAN can do once it's trained with style transfer to match Daggerfall.

Post Reply