AI Upscaled Textures

Show off your mod creations or just a work in progress.
Post Reply
User avatar
King of Worms
Posts: 4752
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

Damn those overlays.. :ugeek:

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Those overlays look fantastic--I've gotten this process working really well over the past week so if applying it to generate enhancements for something like that is beneficial feel free to hit me with anything you'd want processed and I'll hook it up.

on that note:

I went ahead and created preliminary Python classes for processing bulk image assets, and processed the rest of the sprite portraits (that I have) to human photos after spending a little bit of time adjusting params. (The original sprites are from a download of old/partial assets that I found here on DF Workshop forums, may have been this thread I can't remember).

Here is the link to the ones I've processed--photo outputs are in root folder, "comparison" folder has a side-by-side pf photo-output with the original:
https://mega.nz/file/FbR0hJ4R#VHN-HeSeC ... lf9xtxEv1A

The biggest issue is headwear (hoods mainly) being turned into hair. However that can be resolved without too much hassle (short-term just gonna use dlib and opencv to stitch the face back into to original image, until I can add a layer to detect the type of headware from the original portrait and generate a photo realistic addition of it to the existing photo renders ).

Other than that, the sprite-portrait-to-human-photo functionality is basically done--a newer or more complete/polished set of sprites is highly likely to generate even better output, but it's worked well enough to handle a pretty decent amount of variation using the partial/older portrait set I found on the forums here.

Here are some mosaics of what's in the link above--for each NPC, left is the sprite i used as input, right is the output:
example1.jpg
example1.jpg (142.18 KiB) Viewed 2680 times
example2.jpg
example2.jpg (146.32 KiB) Viewed 2680 times
Additionally, I've finished up getting a handle on and integrating StyleGAN2-ADA to the pipeline--just gotta get the right combo of weights and explore art styles for retraining. 3 thumbnails for each sprite here. Left is the OG sprites, middle is human render, right is a preliminary test on style transfer from human render to a Renaissance Art style :
example3.jpg
example3.jpg (81.27 KiB) Viewed 2680 times
This is just an example, and was very *lightly* processed (only 100 iterations, more like 500 will be a lot better), but I have the ability to blend this style with other existing pretrained models without having to retrain. The old beater of a K40 rig I mentioned in another post is still training on a couple of datasets for a handful of other styles I couldn't find weights for, but there's already a ton of other publicly available weights that are high quality and ready to be experimented with. Going to be testing out several comic book weights blended with Anime to see what I can generate in the next couple days.

Again these are just initial results since I've gotten the pipeline and some preliminary code and containerization infrastructure hooked up--the resultant paintings aren't something intended to be useful yet, just wanted to demo that the "guts" of automating this are in place. (multiple different tensorflow-gpu + CUDA toolkit environments on conda/k8s until I can reimplement and unify dependencies in my own code for the models. Using another container for async mgmt of generic image prep and parsing)

Hoping to have some pretty wild content to share in the near future--got a good bit of the hard part out of the way. I'll keep updating as I have new things to share--if there is a good sample of portrait assets anyone would want me to test turning into human photos/art, I'll be doing a lot of test driving in the next several days and can post the results back here or via dm on request.

User avatar
Merlkir
Posts: 25
Joined: Thu Dec 28, 2017 12:23 pm
Contact:

Re: AI Upscaled Textures

Post by Merlkir »

That is way cool!
There's some information lost in the style transfer, like the renaissance style is probably trained mainly on white people, so the skin gets lightened in general. And some of the facial expressions become less interesting, we lose makeup, accessories and clothing.
But hey, all of that could be manually added back in.
This is cool.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Yeah even heavy processing for a single image for the Renaissance style seems to turn everyone into a white person so the weights for that are broken out the box (not usable). It's possible to iterate over it enough that enough times that the familiarity comes back on some of the characters but the dataset it was trained on is inherently flawed.

Good news is that I stumbled on a huge pile of high quality weights from comic books and anime, as well as a a ton of others from more modern artwork that look much more promising. I haven't tested "blending" multiple styles from different pretrained weights yet either--it was a huge surprise to me that was even possible. About to go down the rabbit hole this weekend and hopefully emerge with something interesting and useful.

User avatar
King of Worms
Posts: 4752
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

Damn Ive left my flat so I dont have my data here. I would sent you all the HD portraits. And maybe exported all tge original low res portrats from game to experiment on.

Maybe some MOB sprites to test the results?

Ill be back in a week or so..

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

No worries--I have plenty of work to do to get the remaining process for transfer learning in StyleGAN2-ADA tested and in my own code with documentation (blending of multiple visual styles using the adaptive discriminator variant of StyleGAN2).

I eventually want to build out an extendable Python API for several variations of a handful of use cases for StyleGAN2, which will take some time. Much of the "demo" resources out there for this tool are just like "look what I can do, follow me on Instagram" from Medium articles that link to dead repos without a ReadMe, and not "here's a community-built FOSS library with docs under the MIT license you can leverage to produce a deliverable".

An elegantly simple, free and open source interface in Python to resources like StyleGAN2, that provides easy installation and access to a modular variety of bleeding edge Machine Learning tools is sort of a goal I've had for some time now. If someone could slap a GUI on it or make it into a drag and drop plugin for a game editor it'd really bring the power of these and other computer vision models to the masses.

All that being said, I am going to work on a complete refactor of both the pixel2style2pixel example code and the tensorflow-gpu implementation of StyleGAN2-ADA that I modified for use in the demos I've posted, into a library of modular and extendable tools in Python for enhancing 2D sprites. I'd like to make a public GitHub repo for it, hook up some sort of build automation, sort out branching, hook up documentation and really get it in an acceptably easy to use format before all of that. Ideally making the models replaceable, while keeping the interface intact as new technology makes these techniques obsolete.

I'm still very much in the prototyping phase for this, just wanted to share some greater aspirations on making disruptive tech more readily available to artists and game developers. Art and Science are as fundamentally inseparable disciplines as Chemistry and Physics. Avante Garde and the bleeding edge are one in the same when experience drives perception. There is a lot of opportunity for the Daggerfall modding community to demonstrate that point as new technology emerges.

TLDR I'll keep updates posted as progress continues--no hurry on the assets as I have plenty to do in the meantime.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Also, wanted to mention that if I make a public GitHub repo for any code I used here, that I absolutely will not share or redistribute anyone else's asset datasets or other artwork (modified by me or not) unless expressly given permission to. In fact if I need to take down the link to the example dataset of the photorealistic output from the old/partial pack I found here let me know. I am assuming it is in the clear as the link for the original data is still up and the data itself is old/unused in any mods afaik.

I'd demo the repo with unrelated images generated by another GAN or taken from totally unrelated games etc. Just wanted to be clear I'm aware of the wishes to not redistribute asset datasets from anyone else's mods or creative work, should they be shared with me collaboratively in the future.

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: AI Upscaled Textures

Post by MasonFace »

Fantastic stuff. I very much appreciate what you, and people like you do to make ML applications more accessible to the rest of us. I remember wrestling with the ESRGAN and SFTGAN scripts in a command line interface and it isn't fun, especially when you accidently update a library or something and it causes the whole thing to stop working (learned about Conda environments way too late). :lol:

If you end up using my upscaled portraits, I have no problem with your distributing. KoW may have a more recent copy of them more readily available and probably more cleaned up/touched up than my set. Obviously he'd have to give the consent for that version of them.


Thinking about this longer term, this could be a great utility for creating brand new character portraits that match the style of Daggerfall. I think it would be very apropos to make character portraits for InterKarma and some of the OG developers. I mean if the user can input a photo, the StyleGan should be able to reconstruct that photo in the Daggerfall style, right? This could open up so many great possibilities for Modders!

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

That is correct--StyleGAN can be used for a very broad range of things too, not just 2D but also to generate "neural textures" from a couple of photos (front and back) that can be UV remapped to generated meshes also from the same input image.

There are a couple of models out there for generating pretty amazing 3D meshes of humans from an input photo, so one idea that has actually seen pretty good results is to take those "neural textures" generated by StyleGAN2, and take the 3D mesh generated from the same input images and rasterize a model from them in Pytorch using efficientnet-b7. I have this code working locally for generating random textures and meshes, but for inference on creating a new texture and mesh from a new input photo, I am having to set up a colab environment (inference takes about 40 seconds but consumes a massive amount of GPU memory);
https://github.com/saic-vul/style-people

Here's a video of the above technique, I am working to retrain pixel2style2pixel to do full body conversions (downloaded a huge dataset for this from Google open images), then use the Daggerfall photorealistic renders in the neural textures/meshes to bring them into 3D assets that are fully rigged down to the finger bones:
https://youtu.be/fsuFmpasne8
Last edited by bits_n_giggles on Sun Jun 27, 2021 9:29 pm, edited 1 time in total.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

When they are rendered into 3D this way, the clothing is something that can be isolated and swapped out/removed too as it is a separate rendered object from the body.

Post Reply