3D Humanoid Model Replacer

Discuss modding questions and implementation details.
Hermit
Posts: 4
Joined: Sat Feb 06, 2021 4:51 am

Re: 3D Humanoid Model Replacer

Post by Hermit »

MasonFace wrote: Mon Dec 14, 2020 9:58 pm No, I'm afraid not. I've been so busy with other projects that I haven't had a chance to try to compile UMA for the newest version of DFU.

After Hrafni started making high quality humanoid models, I kinda felt less enthused about going through the trouble integrating UMA into DFU.

I was going to help Hrafni implement his models into the game, but I don't think he's been around lately, probably tied up with IRL stuff if I had to guess.
I can't tell you how sad I was to read that you've decided to stop working on this mod... Could you please post what you've got so far on https://www.nexusmods.com/daggerfallunity?
And please give permission to someone to continue working on it?

PeterSchultz
Posts: 1
Joined: Tue Feb 09, 2021 3:01 pm

Re: 3D Humanoid Model Replacer

Post by PeterSchultz »

If you could update us on this 3D model, I would like to continue/help you with the mod.

If you gave up on it, and give me permission to continue your work you will be credited as the creator, otherwise contact me so I can help you with it.

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: 3D Humanoid Model Replacer

Post by MasonFace »

Not sure who these last two posts were addressed to.

Concarne was the originator of this thread and he was replacing the 2D villagers and some MOBs with skinned 3D models.

Shortly after, TheLacus implemented modding hooks for a proper implementation of 3D model replacement of mobile billboards.

Later on, I tried replacing the 2D villagers with configurable humanoid meshes generated at runtime with UMA. This involved compiling UMA into a DLL, but I couldn't get version 2.0 to compile correctly, and 2.10 required the project to support .NET 4.0 or greater, which DFU did not at the time. I haven't gone back and tried it again since then.

After that I implemented my SpriteWrite utility in there, but it was really buggy when two characters got close together and I was never able to solve this issue.

What was shown in the (NSFW) video I posted was all in the Unity editor. I couldn't get the whole thing packaged as a mod because DFU didn't support .NET 4.0 at the time.

As for preserving my work, there's really nothing for me to upload. Anyone can download UMA from Github and compile it into a DLL.

Then follow this thread on how to load the DLL as a mod.

They could then follow TheLacus's tutorial for replacing a 2D Mobile with a 3D Mesh to get the character in game.

I've backed up my SpriteWrite Utility as part of Hrafni's Playground test scene if anyone wants to try to incorporate that and fix it up.

If you're just wanting the 3D models that Hrafni (Hrafnyx) made, then you'll have to wait for him to contact you.

Anyhow, I hope someone will pick up the torch and make something great out of it. I just don't have the time nowadays.

l3lessed
Posts: 1398
Joined: Mon Aug 12, 2019 4:32 pm
Contact:

Re: 3D Humanoid Model Replacer

Post by l3lessed »

Does DFU support the net framework to pick this up again?
My Daggerfall Mod Github: l3lessed DFU Mod Github

My Beth Mods: l3lessed Nexus Page

Daggerfall Unity mods: Combat Overhaul Mod

Enjoy the free work I'm doing? Consider lending your support.

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: 3D Humanoid Model Replacer

Post by MasonFace »

I think so. Seems like DFU supports .NET 4.0 after it was moved over to Unity 2019 LTS.

Edit: Removed my renege after InterKarma confirmed my original statement.
Last edited by MasonFace on Thu Feb 11, 2021 2:46 am, edited 1 time in total.

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: 3D Humanoid Model Replacer

Post by Interkarma »

I can confirm that DFU uses Mono 4.x in current master. :) We switched over around the same time we changed to Unity 2019.4 LTS.

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: 3D Humanoid Model Replacer

Post by bits_n_giggles »

Sorry for showing up out of nowhere here, but I saw this thread and had a question for either Interkarma or any other developers here experienced with replacing these types of sprites with 3D models.

Would it be beneficial if someone were to automate the generation of 3D humanoid models for this, using the original sprites, through a convolutional/GAN pipeline? I am under the impression the primary challenge here is manual creation of 3D models--if that is not the case or if it isn't the only roadblock, my apologies ahead of time. I have been lurking progress/details on this this specific topic, and wanted to get as much of an idea of the scope involved prior to working on anything in my spare time.

Starting with humanoid NPCs, it appears there is enough information for me to get started on recreating the work that's already been done here in order to get a handle on the technical details of the process--hoping to run through most of this in the next few days (btw, thank you for the info on this process too).

What I am considering trying, is starting with the original sprite images, running them through super-resolution to upscale them as much as possible, then testing the demo from "PIFuHD" here: (has links to code repo, google collab for an online demo, the paper, and video of results: https://shunsukesaito.github.io/PIFuHD/

There's a good chance the demo from that paper/repo won't produce production ready results from 2D sprites without an intermediate network for preprocessing/generation of new images, but I'm in the process of testing that now. However, if I can even just take public domain images or use Nvidia Style GAN to generate photorealistc images of humans based on the 2D sprites--if generating reasonably "acceptable" models as a "starting points" is of any use, I'd definitely be open to learning and attempting to automate the remaining steps involved with preparing them as assets ready for use in game.

Again, my apologies for the lack of knowledge specifically on DFU modding (working on getting up to speed on that). I have plenty of development experience but almost all of it pertains to ML automation pipelines and CV. Thanks for everyone's work here--I've been pretty inspired by what I've seen DFU become in recent years, and would love to help if anything I've mentioned is applicable. 3D sprites would really add another level of feeling like a modern game.

I'll try to follow up with results of processing a couple sprite images through the pretrained network I posted above as soon as possible (even if it's a flop/garbage. Thanks again!

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: 3D Humanoid Model Replacer

Post by MasonFace »

I am under the impression the primary challenge here is manual creation of 3D models
Yes, I'd say that's the biggest hurdle.
What I am considering trying, is starting with the original sprite images, running them through super-resolution to upscale them as much as possible, then testing the demo from "PIFuHD" here: (has links to code repo, google collab for an online demo, the paper, and video of results: https://shunsukesaito.github.io/PIFuHD/
I tried this about a year back using the sprites that I upscaled using ESRGAN -> GigapixelAI -> TopezSharpen AI, if you want to take a look: https://forums.dfworkshop.net/viewtopic.php?f=14&t=3848&p=44928#p44928. The results weren't good, but I don't have any experience training ML models, nor do I have the hardware for it.

King of Worms may be able to provide you with some samples of the highest quality upscaled and polished sprites we have available if that would give you a better starting point.

I hope this helps! And good luck!

User avatar
bits_n_giggles
Posts: 23
Joined: Fri Jun 18, 2021 12:37 am
Location: United States
Contact:

Re: 3D Humanoid Model Replacer

Post by bits_n_giggles »

Thanks for the quick reply. After more research I've found the latest model from Nvidia that does exactly what I think we're looking for called GANverse3D--it apparently works well enough that it's included in a new toolkit from Nvidia that's in open beta called Omniverse.

A bit about GANverse3D here:
https://www.awn.com/news/nvidia-unveils ... omatically.

Omniverse open beta download from Nvidia:
https://www.nvidia.com/en-us/omniverse/

A DL for the latest/greatest sprites would be ideal (probably needed as png or raw image flats). How might I go about reaching out to King of Worms? I am quite new here (just signed up today lol) so DM feature isn't unlocked yet; no biggy if I need to follow procedure and build up a bit more activity on the forums first etc.

Thanks again!


...
--Update/Edit:

Looks like that tool isn't actually released yet, and Omniverse is just a GUI framework for external Nvidia rendering tools and code that will include it in the near future.

There is a placeholder for Nvidia Toronto's GitHub repo for GANverse3D here: https://github.com/nv-tlabs/GANverse3D

and per one of their blogs from a couple months ago:
"The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June."

They are probably finishing up demoing the thing prior to public code release (I'm hoping). Looks very promising for DFU, this will give me time to familiarize myself with the modding process in the meantime.

Edit2: "Code" section in the below link says "coming soon" so yeah looks like we just gotta wait a few weeks. Results look decent, there are a couple of other tools it looks like they used to automate cleaning up the models and animating them in Omniverse as well, so GANverse3D is likely just one part of a parameterized processing pipeline that starts with just a big pile of 2D images and ends with polished/rigged/animated models. https://nv-tlabs.github.io/GANverse3D/

model from their demo:
https://blogs.nvidia.com/blog/2021/04/1 ... omniverse/

sorry to spam so much Nvidia crud--not really a huge fan of them as a company but this seems to fit the use case. We'll see, they have a tendency to over hype sometimes.

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: 3D Humanoid Model Replacer

Post by King of Worms »

Well, you dont reach for King of Worms, he reaches for you. When the tools are ready, reply to this msg and I will get back to you with some set of png images of various mobs.

Post Reply