AI Upscaled Textures

Show off your mod creations or just a work in progress.
User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Yeah exactly what Merlkir mentioned is what I'm currently working on--I'm actually training the "painterly style" model you mention as we speak on the original sprite images.

It's hard to explain but the pretrained model that was used in order to create a "sliding scale" between the input photo back to the original sprite's art style--that model was only trained on human photos, so it changes the original sprite too much to look like a human photo when latents are extracted from it using this model.

That will be fixed by adding into this process, the fine-tuned model that I'm training above that can generate entirely new sprites--I have some testing to do to make sure the learning rate and other params are good but I've figured out transfer learning which makes this an order of magnitude faster/easier (having to retrain if results suck is a more realistic option now).

In the video I posted, the progress bar can be thought of as a slider in a gui, where left is the amount of the photo render to include, and right is the amount of the original sprite to include. Along that sliding scale, you have "latents" which are like generated variations of encodings of an image that can be "mixed" with latents from other images. I calculated 1000 latents for the original sprite and it's photo render, then interpolated each of them against each other respectively, which is what that video is--a traversal of the visual representation of the interpolation of sets of latent vectors extracted from those two images.

However, that model was only trained on real photos. That's all it knows how to generate. So when it calculates latents for the original sprite image, it all but turns the sprite into a photorealistic human as that's all it knows how to generate. DLib, which is used in most of these examples to find faces and help align them, still detects it as a human face, but StyleGAN2 has to then render an encoded representation of what each latents "face" looks like across a gradient scale of variation. Since the generator only generates humans, that's what we ended up getting too much of here.

The solution is to use an additional model that generates sprites along with this process, to be able to provide the original art style of the sprites as it is needed to generate accurate latents. I have working code on deck to do this, just gotta finush fine-tuning a portrait dataset that can generate sprites that look like the originals, dial-in the interpolation, and that'll have the full "pipeline" ready to go.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Also, I can certainly just keep using the existing set of sprites, but I am ready for a good representative sample of the latest/greatest assets to use for retraining the model.

King of Worms--if you're willing to provide that, I'll follow up accordingly. I'll be offline after this post for about 5 hours or so, but will be back online this evening (about 7PM US, EST).

I can probably make it work with 340 images, but the more the better. An even distribution between the appearance of sprites in the dataset isn't necessarily required, but so long as the dataset isn't like 60% Dunmer with a hodgepodge of all everything else it should work fine.

Also if you just have a huge dataset of sprite portraits and it's a bit of an effort to produce a curated dataset, I've got an entire Narnia of my own image dataset prep tools, so don't spend too much time manually putting anything together if it's easier just to provide a raw image dataset.

Data wrangling is about 90% of my book of spells, the other 10% is the arcane language of machine learning. I'm not a traditionally trained Wizard of computer science, but I've been an acolyte of automation for some time. So if there are any tedious tasks that come with the territory I can likely summon a bot to help handle it (∩`-´)⊃━☆゚.*・。゚

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Finished fine-tuning an ffhq variant via transfer learning, and integrated it into the existing style mixing--initial results look pretty darn good.

Image

If the style here isn't ideal let me know as it's able to be "dialed in" on the fly and for different features, on an automatic basis for each sprite. Other than that, it's ready to upgrade a huge batch of sprite portraits now.

User avatar
King of Worms
Posts: 4752
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

Looks great but its based on a AI upscale. Can it work with the original as a source? :lol:
395-0.PNG
395-0.PNG (2.6 KiB) Viewed 2288 times

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

King of Worms wrote: Thu Jul 15, 2021 10:13 am Looks great but its based on a AI upscale. Can it work with the original as a source? :lol:
395-0.PNG
I can add an upscaler to the pipeline, but didn't, as I was designing this on enhancing the effort that was already put into upscaling (on upscaled sprites) to apply a higher quality art style.

I can certainly redo that part of it, and probably end up with better results than what I'm using for input now, but it's a marginal benefit for the effort (if it's already been done, and better images already exist than what I used to get these results).
Last edited by bits_n_giggles on Thu Jul 15, 2021 2:24 pm, edited 2 times in total.

User avatar
King of Worms
Posts: 4752
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: AI Upscaled Textures

Post by King of Worms »

Fair enough and makes sense 👍
These upscales u base it at are the best AI upscales available and I partially use them in DREAM mod as well.

Where they dont excel is with the hooded figures. The hood is not that good. I fixed that in my mod. Also some portraits in my mod were based on sub par BRZ upscales but those were further enhanced by various artists and me. And those Id say look better than AI. Problem is Ive applied shaders on them and that might mess up your AI if u decide to use my images.

The VIP portraits (those with golden frames) are mostly completely redrawn in my mod.

Still... I can make a pack and send it your way and you might take a look. I think some of my portraits will provide better base maybe.

But it might introduce inconsistency in terms of style, no idea on how that will translate towards the final AI results

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

What you mention with the hooded figures is a good point, and actually gives me a few ideas.. It also is making me consider turning this into a Unity plugin with a GUI, instead of just using it as a one-off data processing task.

StyleGAN2 can absolutely address "accessories"--this will be a challenge for me to figure out, but I have seen it address exactly what you've mentioned here and then some.

Generating "style-configurable" dimensional latents can allow you to add and seamlessly modify hoodies/robes/hats/jewelry/clothing/tattoos/piercings/hair color/basically any isolated element of accessory or appearance imaginable, as well as the ability to generate massive sets of standalone accessories or attributes that you can choose from to then fit to a humanoid character in another image, and seamlessly/automatically transform that accessory/attribute to that characters art style.

That is to say, StyleGAN2 is more than capable of facilitating literal drag and drop accessory/attribute "outfitting"--that is easier said than done but it's not much more than elbow-grease in terms of coding to make that happen. The model can do it, you just need an interface, which I can develop (I don't have to do a dissertation on a new model or anything lmao).

When we can manage to get it that far, that opens up an entire Narnia of another wild territory--geberative animation.

StyleGAN2 isn't limited to single images--it wasn't intentional but as you saw in the video example or the "morph", the characters mouth shut as it interpolated between another mouth dimension that was shut.

That might not seem like much but you can isolate the dimension for morphing just the mouth. Or just the legs. Or just the eyes etc.

This allows you to generate any movement animation on any latent from any model you want.
This is the pathway to text to speech animation, blinking and breathing animations, hair animations, (and in non human networks) hyper realistic fire and water animations, explosion animations, electrical animations... (You get the point)

With all of that considered--perhaps turning this into a Unity plugin with an easy to use GUI would be more desirable? It seems like it would be of more value to provide mod developers tools to access easy creation of the assets.

I have very little C# or even Unity experience but I have over a decade of Python and C/C++ chops. I have no idea how to make a Unity GUI plugin but I can absolutely make a backend out of my existing code to connect sliders, drag n drop and other controls to.

If someone here already has experience in Unity plugin development, I would love to collaborate on that. I can learn it myself too but an expert on that would allow me to focus on heavy lifting that'd be needed for the backend for it.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Also, wanted to put it out there that the value I'm getting out of this project is the skill development of making it work. It's easy to stay motivated because I am a megafan of TES and have always wanted to see Daggerfall brought up to the future. It would take time to make something like what I mentioned in the last post, but I wouldn't bail on it if there's actual interest in it.

This was originally just the starting point for bringing sprites into 3D too, so I may already have what I need to start work on that (if the existing 2D sprites in DREAM already look better than these). I'll go ahead and process all the sprites I have on hand, and probably go ahead and generate about 40,000 or so totally new/unique sprites in the resultant quality above (edit: I'll refrain from sharing a generated set, idk how folks feel about that being distributed so I'll ere on the side of caution)
Last edited by bits_n_giggles on Thu Jul 15, 2021 4:29 pm, edited 1 time in total.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: AI Upscaled Textures

Post by bits_n_giggles »

Oh I almost forgot--

The 3D conversion requires a full body image as input, so I actually do need a set of full body sprite images if that's doable. The matching portraits would be super helpful too, around 340 or so sprites would do the trick. Let me know and I'll follow up as needed

User avatar
MrFlibble
Posts: 410
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble »

bits_n_giggles wrote: Thu Jul 15, 2021 4:25 pm The 3D conversion requires a full body image as input, so I actually do need a set of full body sprite images if that's doable. The matching portraits would be super helpful too
If you mean the villagers, it's a bit tricky because their walking animations are rendered at a lower resolution than the two standing frames when they're facing the player.

Post Reply