Sorry for showing up out of nowhere here, but I saw this thread and had a question for either Interkarma or any other developers here experienced with replacing these types of sprites with 3D models.
Would it be beneficial if someone were to automate the generation of 3D humanoid models for this, using the original sprites, through a convolutional/GAN pipeline? I am under the impression the primary challenge here is manual creation of 3D models--if that is not the case or if it isn't the only roadblock, my apologies ahead of time. I have been lurking progress/details on this this specific topic, and wanted to get as much of an idea of the scope involved prior to working on anything in my spare time.
Starting with humanoid NPCs, it appears there is enough information for me to get started on recreating the work that's already been done here in order to get a handle on the technical details of the process--hoping to run through most of this in the next few days (btw, thank you for the info on this process too).
What I am considering trying, is starting with the original sprite images, running them through super-resolution to upscale them as much as possible, then testing the demo from "PIFuHD" here: (has links to code repo, google collab for an online demo, the paper, and video of results:
https://shunsukesaito.github.io/PIFuHD/
There's a good chance the demo from that paper/repo won't produce production ready results from 2D sprites without an intermediate network for preprocessing/generation of new images, but I'm in the process of testing that now. However, if I can even just take public domain images or use Nvidia Style GAN to generate photorealistc images of humans based on the 2D sprites--if generating reasonably "acceptable" models as a "starting points" is of any use, I'd definitely be open to learning and attempting to automate the remaining steps involved with preparing them as assets ready for use in game.
Again, my apologies for the lack of knowledge specifically on DFU modding (working on getting up to speed on that). I have plenty of development experience but almost all of it pertains to ML automation pipelines and CV. Thanks for everyone's work here--I've been pretty inspired by what I've seen DFU become in recent years, and would love to help if anything I've mentioned is applicable. 3D sprites would really add another level of feeling like a modern game.
I'll try to follow up with results of processing a couple sprite images through the pretrained network I posted above as soon as possible (even if it's a flop/garbage. Thanks again!