AI Upscaled Textures

Show off your mod creations or just a work in progress.
MrFlibble
Posts: 107
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Tue Jan 08, 2019 12:06 pm

MasonFace wrote:
Sun Jan 06, 2019 10:51 pm
@MrFlibble
So when you "soften" the image using xBR, are you using xBRZ Scaler Testing Tool with the Scaler = "3" (xBR), and Scaling Factor = 1 ?
Here's how I do it:
  • load image in xBRZ Scaler Testing Tool, scale to 4x the original size (Scaling Factor = 4), with either xBR or xBRZ
  • save the new large image, then load it in GIMP
  • apply pixelise filter with 4 pixel radius
  • scale image back to the original size (or 25pc of current size) using nearest neighbour
  • export the resulting image
MasonFace wrote:
Mon Jan 07, 2019 6:15 am
Got a few test MOB animations upscaled. They all worked well using Mr. Flibble's xBR softening pre-upscaling technique in tandem with SFTGAN (for the Orc and Gargoyle) and ESRGAN (for the Vampire).
That's pretty cool!

MrFlibble
Posts: 107
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Tue Jan 08, 2019 6:18 pm

So I tried the Manga model myself. Here's a comparison of the same image run through waifu2x (Y-channel scaling) and Manga (ESRGAN):
Image
Image
The pre-rendered models look really good with the Manga model. (But I think that it might have been trained on JPEG images and reproduces some JPEG artifacts?)

User avatar
King of Worms
Posts: 1621
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow
Contact:

Re: AI Upscaled Textures

Post by King of Worms » Tue Jan 08, 2019 6:25 pm

Yes the pre rendered models in a 2nd screen are crisp, very nice

Kaucukovnik
Posts: 15
Joined: Tue Jan 08, 2019 4:23 pm

Re: AI Upscaled Textures

Post by Kaucukovnik » Tue Jan 08, 2019 7:04 pm

Agreed, the pre-rendered ones look perfect. I'd have believed it's some Bethesda-originating source graphics you managed to dig out. Well, especially older CGI artwork does have more rigid and predictable lighting than photos or paintings, so it makes sense that it's easier for an AI to interpret.
If this method works this well with all the animated characters, I'd consider them done. It will take some time before any other area of Daggerfall artwork reaches comparably nice and purist-friendly facelift.

My pet peeve with upscaling algorithms and AIs is that they tend to interpret intensive flat detail as massive bumps or recessions. Like the gold ornaments on the dress. And....what's wrong withe her FAAAACE? :)

MrFlibble
Posts: 107
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Tue Jan 08, 2019 7:19 pm

Here's an "SVGA" mock-up of the character stats screen (from official Bethesda screenshot):
Image
I used the Manga model and applied Gaussian blur at 1 pixel radius before scaling down to 640x480 with Sinc3 in GIMP, then converted to the original palette with mtPaint.

Here's what the original image would look like when corrected to 4:3:
Image

She looks squinting but the rest of the image is pretty neat. I like how Manga handles natural backgrounds, tree foliage and stuff. It's almost as if that background was painted in higher resolution manually by an artist.

I'm noticing one problem with the model, it seems to rather often exaggerate vertical lines, e.g. the small finger on her right hand is visibly detached from the hand. Same stuff with the Conjurer's ear on the Warcraft briefing screen:
Image

User avatar
VMblast
Posts: 469
Joined: Wed Mar 29, 2017 12:22 pm
Contact:

Re: AI Upscaled Textures

Post by VMblast » Tue Jan 08, 2019 8:17 pm

WOW! Awesome work! Cant wait implemented MOD! :D

User avatar
MasonFace
Posts: 262
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Tue Jan 08, 2019 9:20 pm

@MrFlibble
Ah, so I wasn't even doing your method right. I was just using straight xBR output as the input for those MOB's SFTGAN and ESRGAN upscales... looks like when you did it, it gave very good results on the edges!

Also, I'm glad to see you've got the AI scripts up and running on your machine. Now we can double-team this and get it done much quicker! Not to mention we can both experiment with different pre and post filters to tweak the results.

----------------------
@All
I'm trying to come up with a way to easily batch all the images that need to be upscaled so we can ram all this through the pipeline quickly. What we need is something that can remove the alpha layer and replace it with a specified color so we can easily swap it with alpha after the upscale is done. I've tried using ImageMagick, but I am an absolute novice at it and can barely get it to do anything. I've also tried using BIMP for GIMP to batch process the images, but it doesn't look like "Flatten" does anything when I try to use it... it still keeps the alpha layer for some reason. Anybody got any suggestions?

----------------------

On a related note, until I get a method down to batch all the remaining sprites, I've been working on upscaling the portraits in the meantime. Below is an example with the original on the far left, then the xBR 4x upscale, then the SFTGAN output with an unmask sharpen filter applied, then the far right is a quick manual touchup (about 5-10 minutes) adding some shadows and cleaning up some edges.

Image

User avatar
pango
Posts: 1091
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: AI Upscaled Textures

Post by pango » Tue Jan 08, 2019 10:37 pm

MasonFace wrote:
Tue Jan 08, 2019 9:20 pm
@All
I'm trying to come up with a way to easily batch all the images that need to be upscaled so we can ram all this through the pipeline quickly. What we need is something that can remove the alpha layer and replace it with a specified color so we can easily swap it with alpha after the upscale is done. I've tried using ImageMagick, but I am an absolute novice at it and can barely get it to do anything. I've also tried using BIMP for GIMP to batch process the images, but it doesn't look like "Flatten" does anything when I try to use it... it still keeps the alpha layer for some reason. Anybody got any suggestions?
Replace transparent pixels with green (#00ff3c) pixels with ImageMagick:

Code: Select all

convert -compose Dst_In -background '#00ff3c' input.png -alpha remove flat.png
And back (will some color fuzziness as an example, but that's not required):

Code: Select all

convert -transparent '#00ff3c' -fuzz 5% flat.png back.png
It seems there's some support for batching, but in this case I'd simply use a for loop, or whatever your favorite command-line shell provides...
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

MrFlibble
Posts: 107
Joined: Sat Jan 27, 2018 10:43 am

Re: AI Upscaled Textures

Post by MrFlibble » Wed Jan 09, 2019 1:21 pm

In the meantime, I came up with what seems like a good idea for high quality data to train a model for sprite scaling.

First off, I think that it would be reasonable to discern hand-drawn and pre-rendered sprites. For the latter, you can use authentic (relatively) high-resolution renders that have been released for various games back in the 90s.

Off the top of my head, here are a few examples:
Spoiler!
Warcraft II: Tides of Darkness
ImageImageImage
StarCraft
There are official wallpapers with large renders and large hand-drawn art for some units and characters.

Might and Magic VII
ImageImageImageImageImage
Heroes of Might and Magic II (hand-drawn)
Image
Rise of Nations
ImageImageImageImageImageImageImageImageImage
Tomb Raider
ImageImageImage
I propose to add black background to images which come with transparency, since this seems the most useful background colour for processing Daggerfall sprites.

There are more renders of this kind known at this point, but many are in JPEG quality, and thus not usable for the training purposes.

Another thing to note is that we need to carefully prepare the low-resolution versions. ESRGAN and similar networks used for photographs are created with the assumption that the LR image input are scaled down, low-detail copies of higher resolution originals, however low-resolution video game sprites are usually not defective, they are complete works of art. So far I've no idea how the MATLAB script proposed by the ESRGAN paper authors produces low-res images. Are you going to create low-res images manually? I have just tried scaling down the Colossus sprite from Rise of Nations to 25% its full size with Sinc3 interpolation and then convert to the same palette as the original GIF, and I'm not sure I like the result:
Image
It has become pretty messy, the red line at the base for example lost its curve that Sinc3 actually tried to preserve by introducing softer halftones.

UPD: Removed those X-COM images, they're too noisy because of dithering.

User avatar
MasonFace
Posts: 262
Joined: Tue Nov 27, 2018 7:28 pm
Contact:

Re: AI Upscaled Textures

Post by MasonFace » Wed Jan 09, 2019 8:11 pm

Replace transparent pixels with green (#00ff3c) pixels with ImageMagick:

convert -compose Dst_In -background '#00ff3c' input.png -alpha remove flat.png
@pango:

Thanks! I looped the code for all images in the folder and it's working great! That will save a lot of time.

I'm thinking that I'll fill the transparency to black per MrFlibble's suggestion and see how that works out.

----------------------

@MrFlibble:
I think that it would be reasonable to discern hand-drawn and pre-rendered sprites.
If you're suggesting two separate models, one for hand-drawn sprites and another for pre-rendered sprites, then you'd be correct in assuming the results would be better for each respective case. The problem is that it would require two massive datasets instead of just one massive dataset to train. For this reason, I've come to the realization that maybe training new models is beyond the scope of this project alone. However, there are bound to be many other communities looking to do something similar for other games of the era; if we were to all pool our resources, we could likely crowd source and aggregate the training data quickly and maybe rent some processing time on a supercomputer (or maybe someone with crazy bitcoin mining rig) to crank out the results over a weekend or something. Maybe some folks who are working on build engine overhauls would be willing to help? Whatever model is produced should work quite well for most sprite based games of the same era, even if their respective artwork were produced by differing means. It theoretically won't produce as good of results as what you proposed, but it will produce much better results than any model we have now.

Also, I certainly don't want to discourage you from trying to train new models. It's just that the more I look at it, the more daunting the task seems. And also the more content I become with the results we're getting with the tools we've already got. I'm really coming back around to the idea that we upscale everything to the best of our abilities and current tool set, then release them. I feel pretty confident there will be some talented artists who would love to polish them up from that point.
Are you going to create low-res images manually? I have just tried scaling down the Colossus sprite from Rise of Nations to 25% its full size with Sinc3 interpolation and then convert to the same palette as the original GIF, and I'm not sure I like the result [. . .]

It has become pretty messy, the red line at the base for example lost its curve that Sinc3 actually tried to preserve by introducing softer halftones.
I wouldn't worry about that too much. If you give it less-than-ideal data on the input, it will still train against the ground truth image and (I think) it will end up making the AI model more robust. After all, you are essentially training the AI how to extrapolate missing data from whatever information it has, however poor it may be. Besides, if we need 1,000 images for training, manually downscaling them would likely be infeasible. :cry:

----------------------

I've made a little more progress on the portraits:

Image

After doing the first couple portraits, I looked ahead and noticed that many of the portraits had repeating features so I decided to step back and re-prioritize the non-hooded faces first, then I can simply superimpose the hoods back over them to quickly finish off the rest. They take a while to do, but it is pretty fun work touching them up and I think they look very close to the original artwork. I think this is especially worthwhile since we can slap these faces onto the NPC flats since the AI doesn't seem to do well with the faces on them.

Post Reply