2D Human to 3D Mesh

Show off your mod creations or just a work in progress.
Post Reply
User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

2D Human to 3D Mesh

Post by MasonFace »

I've been looking for projects that can take in an image of an object, especially a humanoid, and create a plausible 3D mesh of that object. A recent news article lead me to a Facebook project that can do this with surprisingly good results.

Image

Check out the author's GitHub page here.

Well, I think you can tell where this is going...

I tried it out on a sample upscaled humanoid sprite (Male Vampire - 285) and the results were... well, not great, but I do see a lot of potential!



I think higher resolution inputs would help considerably, based on what is mentioned in the project:
Use high-res image. The model is trained with 1024x1024 images. Use at least 512x512 with fine-details. Low-res images and JPEG artifacts may result in unsatisfactory results.
I used an upscaled sprite that was only about 256x512. To make matters worse, I had to remove the transparency, and to do so quickly, I just converted it to a jpeg so I could test immediately which probably caused some artifacts...

@KoW: If you read this, could you send me some high resolution samples from DREAM that I could use for tests? Preferably humanoid characters, maybe a few MOBs (try to pick a neutral pose like frame 3 of the front facing walking animation) and a few good hand-drawn NPCs. If you have time, I'd appreciate it if you could replace the transparency with a plain white background. No problem if you don't, I can remove the transparency when I get home from work.

User avatar
King of Worms
Posts: 4752
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: 2D Human to 3D Mesh

Post by King of Worms »

Hi mate, sure I will do it and send u a link. Gimme one or 2 days, Im not at home. ;)

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: 2D Human to 3D Mesh

Post by MasonFace »

On a slightly related note, I stumbled upon another upscaling AI program called PULSE which takes in a low resolution picture of a face and searches in latent space for a similar high resolution face to match it with using StyleGAN.

I tried it with 4 different Daggerfall NPC portraits, tweaking all settings and modifying the input image, and after all that, this is the best result I could come up with:
TFAC0010.RCI_154-0_32 - Results.png
TFAC0010.RCI_154-0_32 - Results.png (1.36 MiB) Viewed 2235 times
The result is interesting, and at least not horrifying like the last program I tried.

I really don't think the AI knows what to do with such disproportionate eyes, and of course there is an artifact at her jawline.

I tried using some filters afterwords, but it still ended up looking ridiculous.

Because there is too much upfront manual pre-processing effort needed before using the program to get a good result, and the result looks far too detailed compared to Daggerfall style, I don't think there is any use for it in DFU modding. But I thought I would share my results if anyone was interested.


Edit: Just gonna add one more...
Results_2.png
Results_2.png (1.42 MiB) Viewed 2232 times

User avatar
pango
Posts: 3347
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: 2D Human to 3D Mesh

Post by pango »

Also, since the algorithm is looking for the best match in the latent space of a GAN, it takes whatever face expression is in there; Above seem mostly neutral/smiling, which is seldom the expression the original image actually had...
It would require some GAN trained with more varied expressions
Mastodon: @pango@fosstodon.org
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
MasonFace
Posts: 543
Joined: Tue Nov 27, 2018 7:28 pm
Location: Tennessee, USA
Contact:

Re: 2D Human to 3D Mesh

Post by MasonFace »

Second attempt, this time using the Zombie MOB sprite (again, 4x vanilla resolution). Also, this time I'm using PIFu instead of PIFuHD. PIFu is an earlier program which has the advantage of adding texture projection. It also supports multi-view setup so you can get better results by using more angles, but the author isn't able to provide the pre-trained model for it, so I I'm not able to use it to its fullest potential...





Either way... it's something.

If only I could get that multi-view support, then I think the meshes would be very useful.

Post Reply