[MOD] Remastered Friends and Foes

[carademono]

Thanks for your gracious response, KoW. I really do admire your work with D.R.E.A.M. -- it's what inspired me to try my hand at this too! I'd be happy to keep talking shop about the sprites or to work together to make Daggerfall look beautiful.

[MrFlibble]

Anyway, as I've repeatedly clarified, my sprites are not simply the product of the XBR filter. I created a base layer using the XBR, on which I overlayed AI upscaled layers. The XBR base layer gives my sprites their strong contours, sharpens detail, and in particular gives the sprites an excellent illusion of depth. More generally, the base layer keeps the sprites from looking like a blurry mess *ahem*.

I've been experimenting with AI upscales for quite some time, and went through this stage of blending an xBRZ layer with AI upscaled layers. The problem is, they're largely incompatible. You mentioned yourself that xBRZ does not handle gradients well -- the truth is that it does not handle them at all, unlike HQ2x and its variations. And this is very noticeable on large sprites.

Why don't we look at some comparison shots?

I try to approach this in as unbiased way as possible, and TBH both your work and DREAM are a mixed bag in these examples. If we only take the commoners (since DREAM upscales of enemies have no faces at all, so it is not really fair to compare them), in some cases both versions look quite bad to me -- this is about the first two female commoners on your list to be exact -- while for some of the male commoner faces, I'm inclined to like the DREAM versions more, like that Redguard guy you said you did not like. I like it more than your version for sure.

Yes, D.R.E.A.M. achieves smoother color gradients, but at the cost of washing out all the detail and flattening any sense of depth. You might also try using a smoothing filter on your alpha layers so their edges don't look so jagged.
<...>
I could make my sprites look like D.R.E.A.M.'s by running them through a Gaussian Blur filter.
<...>
I've got to agree with you here, your technique achieves smoother color gradients than mine. But... as a consequences, your orc looks like it's made of plastic. I spend a lot of time with five-year-olds; I can recognize a Playmobil when I see one.

I understand that these are your major points of criticism regarding AI upscales. You may be right when pointing out these drawbacks specifically in DREAM's upscaled sprites, but this does not apply to AI upscales in general.

Here I quickly ran a single Orc sprite through several ESRGAN models (largely from the Model Database) with decent results, I wonder if you'll also say that the Orc looks like a plastic toy or not (corresponding model name shown above each image):

4x_ArtStation1000_Details


4x_ArtStation1000_Sharp


4x_CleanPixels


4x_SpongeBob-Reloaded_153000_G


4x_FArtDIV3_Fine

[carademono]

Look, AI upscaling is getting better every day, and these orcs look great! But... they just don't look like how I want Daggerfall to look. They look like Shrek. And while Shrek looks really good, I don't want to play Shrek. But somebody does, so you should run Daggerfall's sprites through these ESRGAN models, package them into a dfmod, and release them. I guarantee you that the sprites would attract a sizable user base.

I've said that I use an XBR filter and Gigapixel AI's Art upscaling mode because I want my sprites to look like paintings come to life. The strong color gradients generally don't bother me, because when the come out right, they look like brush strokes -- and that's exactly what I'm going for. I just put up a preview of my next mod here: viewtopic.php?f=14&t=4822&p=54696#p54696. Some of the faces there still have some work to go (particularly the Nord and Breton women), but it should give you an idea of the aesthetic I'm trying to achieve.

That said, sometimes the color gradients don't look like brush strokes -- sometimes the result is just plain bad. But the answer is not AI upscaling. At the end of the day, in my opinion, the only way forward for the front-facing character sprites is for a human artist to retouch them. The best sprites we have right now, by far, are the interior character flats in KoW's D.R.E.A.M. And that's because they were made by hand by an artist, probably using an upscaled image as a base.

You wouldn't want to hand retouch thousands of sprites, but there actually aren't too many front-facing character flats. About 50 in the villagers / commoners (the 5-0s and 5-1s), and another 32 in the classes (the 15-0s and 16-0s). Some monster flats could also benefit from an artist (I'm looking at you, nymph) but overall they're less essential.

[MrFlibble]

I've said that I use an XBR filter and Gigapixel AI's Art upscaling mode because I want my sprites to look like paintings come to life. The strong color gradients generally don't bother me, because when the come out right, they look like brush strokes -- and that's exactly what I'm going for.

Well, everyone has their own preferences, and the taste for video game art is highly subjective -- something that looks marvellous to me could as well be awful to someone else.

But, I'm trying to be methodical, and I believe that a consensus can be achieved on the methods, tools and their use.

I completely agree with you that AI upscales cannot yet be the ultimate answer and human input from an artist is still required to achieve good results. But, in this scenario, the upscales are still an important underlying layer -- the canvas on which the artist works, to put an obvious metaphor to good use.

There has already been a lot of experimenting in the community with various approaches to AI upscaling. And it has generally been concluded that neither the XBR scaler nor Gigapixel AI are well suited for upscaling pre-rendered sprites. Gigapixel AI is a tool for enhancing photographic imagery, and was never intended for anything else. Whereas ESRGAN has a range of user-trained models which have been specifically designed for enlarging video game art (more on that later).

It is also worth pointing out that Gigapixel seems to produce various artifacts on your sprites, like random black lines on helmets:

These don't look like brush strokes to me, they look like the result of an AI upscaler applied to the wrong source material. (I need to point out that I like these faces, but the helmets ruin much of them for me.)

As for XBR/XBRZ, these algorithms have obvious limitations that are clearly seen in a comparison image like this:

I'd say that the AI upscale (it's 4x_ArtStation1000_Sharp this time) should be fairly close to what the original Mark Jones' model looked like. You can also see that this model does not try to add odd artifacts on the helmet or face, however underdetailed it is. While XBR just smoothed out the pixels without even giving the sprite a correct shape at this resolution (jaggity lines everywhere!).

You say that by blending XBR and Gigapixel, you achieve the desired look of a painting. Well, again most of the lines that could be conceived of as brush strokes are too obviously produced by XBR and are essentially just "thick" pixels. Not sure how this is better than just having the original low-res images.

BTW, there's a couple of ESRGAN models that actually make the upscales look like paintings on their own. I don't even mention that you could use a filter in an image editing suite to add such an effect without resorting to upscales of any kind -- here I took the 4x_ArtStation1000_Sharp guard upscale, upped it to 8x with XBR from Image Analyzer, and applied two filters: oil painting from GIMP and brushify from G'MIC. No sure if this is what you'd be aiming for, but I guess commercial image editing software may have better effects of a similar kind.

Or even better, some ESRGAN models produce results that look like a more advanced version of XBRZ, check it out:

50000_G


charsprite


rebout_interp


reboutblend


lollypop

[carademono]

After sleeping on it, I also decided to revisit this and thre my orc face onto one of your ESRGAN models (upscaled x2 in Gigapixel to give it the same resolution)... and you know what, you're right. The ESRGAN model wins:



Instead of continuing to fight over sprites, why don't we just combine all the best ideas, discard the ones that don't work as well, and create a new round of Daggerfall sprites codenamed MrDREAMastered? And when it's ready we release it as D.R.E.A.M. so that people actually download it?

[carademono]

Just to clarify, that wasn't a rhetorical question. I'd really love to form a team with you, KoW, and a programmer to be named later to re-upscale the sprites with ESRGAN, add pretty faces to everything, have the engine mix-and-match faces and bodies as I proposed here (viewtopic.php?f=14&t=4822, and release it as an update to D.R.E.A.M. What say you two?

[MrFlibble]

I'm glad you liked the ESRGAN results after all!

Even though I've followed the ESRGAN trend from almost its inception in late 2018, it's more of an on-and-off relationship mostly now. I haven't attempted any large-scale projects myself and generally limited my curiosity to hopefully systematic testing of models on various single images, because my NVIDIA card model does not support the CUDA capabilities required for ESRGAN, and running it on CPU is much slower.

So while I can try out some of the models -- I'm looking at 8x stuff right now because I realised that ESRGAN actually does not take more time and probably system resources to upscale to 8x compared to 4x -- but hardly run a big batch of sprites to quickly produce usable results. Maybe MasonFace will help in this department.

I should say that AI upscaling made some huge leaps over these several years. The 8x models I have just checked produce some insane results (possible nightmare fuel though!), take a look:

Spoiler!

8x_HugePeeps_v1 by DinJerr


8x_HugePaint_v1 by DinJerr

More images here:
https://imgur.com/a/cztuM8u

If you want something more painted-looking then maybe Glasshopper's MS Unpainter: sample image.

Separately, there are models that are designed to work with faces, here's a couple of samples. For these, I first ran the images through a 1x dedither model which I cannot find links to in the model database right now, then used the 8x_NMKD-Faces_160000_G model from here.



I interpolated this NMKD model with DinJerr's HugePeeps at alpha = 0.1 (meaning 90% NMKD and 10% peeps), and on dedithered images it seems to get a slightly better result with hair:

[King of Worms]

Well yea, ESRGAN wins, theres hardly a way around that. There are various algos etc, also you can use the paint brush effect on the final ESRGAN etc, so various looks can be created. Im up for a cooperation in a sense I can provide all my upscales. I cant spend too much time on it anymore, I just gave it few years already and Im old and tired.

My sprites are ESRGAN base, the front facing frames were further improved by Gigapixel (and eventho MrFlibble said thats a no go, I will show u thats a deffinitely a go go ) and this whole result was than manually cleaned from all errors, the outlines were antialiased (so if you enable AA ingame, the outlines get super smooth - you said I should improve on em - no, just enable AA ingame and they get crisp and smooth ) plus Ive manually added some faces and details. Plus created the emission maps for glowing eyes, whole glowing bodies of fire/frost daedras, fire elemental and their corpses, diversied the werewolf and wereboar looks and most likely tons of other things (like removing magenta tint from every sprite, which for example reverted the Vampire female from that PINK shit color to normal white skinned pale lady u expect the vampire to look like, or the fiery dragon after this correction stopped looking like a pink chicken etc) - really , the input Ive poured to that is vast and I could write a topic about each mob and its corpse... Ive also added custom shading on all the sprites, giving it a bit more of unification, 3d feel and better blend to the scene... just too much to really write here.

So Im quite sure those sprites are not anywhere near to be obsolete. They just cant. Nobody put so much effort in those, simply because theres nobody long enough around focusing on this topic.

So yea, I can provide those as a base for some improvements. Im open to it and Im quite sure its a right way forward. Rest is about discussing on what to do with them, and how to handle the credits and final distribution when its all done.

Or Im open to suggestions on what we can do. I just must state clearly, Im burned out after 5 years, so my input is more about providing results of my work and some opinions and experience.

270_5-5.png (559.33 KiB) Viewed 1988 times

265_0-0.png (287.13 KiB) Viewed 1988 times

260_0-0.png (401.93 KiB) Viewed 1988 times

[carademono]

I cant spend too much time on it anymore, I just gave it few years already and Im old and tired.

Hey, aren't we all at this point? Nobody understands better than me how much work goes into these. I started playing around with sprites as my pandemic therapy; I doubt I'd last five years or even five months before burning out!

If there's one thing I've learned doing this, it's that no approach is going to get good results on every single sprite. We're better off running all the sprites through three different methods and then picking and choosing on a sprite-by-sprite basis. And I totally agree with KoW that Gigapixel is holding its own.

I'm less impressed with these 8x ESRGAN models--they look oversmoothed to me. When I say I hope to give the sprites a painted look, I have in mind a Frank Frazetta painting or old-school D&D box art. That's the aesthetic tradition that Daggerfall belongs to. The Unpainter one does seem to produce the best results of the lot, but I wonder if the winning formula for the commoners might be 4x_ArtStation1000_Details and then upscaled 2x again in Gigapixel?

Interestingly, I think the XBR+Gigapixel combination is still performing best with most (but not all) of the faces. This might be an advantage of Gigapixel's training on photos and its face enhancement feature. You can check those out in the Infinite Villagers thread. But I bet rebout_interp+Gigapixel would do better than that. (The best faces of all, however, are the character creation faces already in D.R.E.A.M.)

I'm heading out to the lake with the family for the rest of the day, but when I get back I'll set up a Google Drive and we'll try some things out! And KoW, I'm feeling super dumb because I made that damn vamp even more pink

[carademono]

Hey all, just got back from the lake and quickly tried out something I'd been curious to do all day -- basically, I tried my recipe but using rebout_interp instead of XBR in the workflow. I also spent (literally) less than 7 seconds removing Gigapixel's upscaling artifacts with the dodge tool. Was interested to try it on the guard especially because he is admittedly my worst sprite... Here's the result of this new process at full 16x size:

Spoiler!

How do y'all feel about this sprite, Daggerfall community? Especially after we put a face on him.