Daggerfall Workshop Forums

That is way too damn cool!

Whoah, there are some breathtaking perspectives opening up for sure here!

I was wondering, just theoretically, could it be possible to process character sprites from Arena with these AI models to make higher-resolution versions that would more the Daggerfall style? NPC enemies also seem to have only the front view, no rotations.

back at home. If there is any requests for either low res original game art exported from game, or some high res DFU art, let me know. Id be quite interested to see what the AI can do to the monsters, if it can turn to 3d or just make the sprites look better. Or the NPCs... I have all art from the game on my HDD.

Just checking back in to keep everyone posted--long weekend away from GPU hardware so I won't be back into it until middle of this coming week, but I am digging into interpolation and latents a bit for mixing styles. StyleGAN2-ada is capable of a lot more than I initially understood, so before I dive into processing a ton of assets I thought I'd take some time to develop a control interface first.

It looks like it may not require retraining anything to get a really good blend of styles across pretrained weights--I have quite a busy week ahead of me but wanted to keep folks posted that I am going to keep at it. It may be a week or two, but I will get some results on style mixing and return back here to try it out on some of the latest/greatest assets.

There is a site called "artbreeder" that is doing exactly what I am trying to do here, and what I am certain I can figure out how to do myself. They essentially have interpolation between latents set up as a "slider" to do everything from add/remove facial hair to blending art styles and more (except they charge for it of course, but luckily they can't do so prohibitively and stop me from making basically the same tool but FOSS). This concept is hinted at in several other repos that I've found on StyleGAN2, and I'm pretty sure I can replicate this at home (and will provide a GitHub repo of the code I use to make an updated and extendable FOSS variant of it):
https://www.artbreeder.com/

Whatever I make probably won't have a GUI, but it'll allow dropping files into place and running a script with args for similar results.

To answer the Arena question, it is likely that it can be enhanced with these techniques as well, I'm just trying to get a handle on how to fully harness StyleGAN2 at the moment. The ability to mix art styles without retraining and just "blend" weights and/or latents is something I feel like I probably need to get a handle on to really get to a point of being able to dial-in an acceptable enhanced art style for Daggerfall Unity.

The site from my last post is definitely proof that the concept of leveraging latents between multiple weights for refined editing of details and style mixing is able to be done, I've just gotta do the footwork to figure out how on earth they did that lol. I have a good start on this, but there are a few mismatched parts (wrong dimensions between input latents and models etc). I'll get it figured out though, just a bit of debugging and a good bit of reading (and a ton of Numpy magic)

Having that piece working, will add the missing level of control we need for the stylization of the photos back into art.

bits_n_giggles wrote: ↑Mon Jul 05, 2021 6:38 am I've just gotta do the footwork to figure out how on earth they did that lol. I have a good start on this, but there are a few mismatched parts (wrong dimensions between input latents and models etc). I'll get it figured out though, just a bit of debugging and a good bit of reading (and a ton of Numpy magic)

Having that piece working, will add the missing level of control we need for the stylization of the photos back into art.

Good luck! This is definitely very awesome and is just way ahead of what we've been doing with AI upscales to this point.

Just wanted to update that I'm pretty close to having this working--not giving up, just been extremely busy.

This is interpolation of a photorealistic render back into the latent extracted from the original sprite.
(from left to right: original sprite, photorealistic render of sprite, mild interpolation towards the original sprite, slightly heavier interpolation towards the original sprite.):


Video of interpolation from a photorealistic render to the latent extracted from the sprite image:
https://www.youtube.com/watch?v=uhDU-VQBIuY


Third from the right in the thumbnails is DARN close, but I'm in the process of fine-tuning an FFHQ pretrained model with sprite portraits to then use for interpolating between the same two images, but also make better use of latents of the art style from the original game. This should result in keeping photorealistic quality but also the original style of the art, only requiring about 300-400 images from sprites for training.

Should have results by the weekend--training only takes about 6 hours for what I need here (transfer learning), I just have to apply a method for incorporating multiple model interpolations as well as one's extracted from input images and we'll be set to apply this to basically any oldschool sprite assets that I can find a pretrained model that's somewhat "local" to the domain of the images we want to enhance.

Also--It's not done yet (this is only on iteration 50 out of 10,000 to go) but consequentially, the model I'm training for the purpose of using it to "blend" the art style back into the photorealistic renders, is also capable of generating entirely new sprite images in massive quantities, in a completely configurable way (need a village-worth of unique high elf NPCs sprite assets? you can parameterize generating that).

On top of enhancements, making every single NPC sprite in the entire game visibly unique, may not be impossible. These are garbage-quality as it isn't anywhere near finished training yet, but it's only going to require 340 sprite portraits to fine-tune the existing FFHQ models. It should be able to generate the exact same quality sprites that it was trained on when it's finished, but also be able to use the interpolation from above between models to not just transfer style but generate new sprites of the same quality as the end-result enhancements that I'm closing in on.

Interesting. I'd say the light blend is a bit more successful than the heavier one. I think the slightly painted feel gets through, but not the cartoony stylization of the face, and it ends up looking a bit zombielike.
Either I'd keep the blend low, or use some other "painterly style" model.

Agreed, 3rd image was the best one