AI Upscaled Textures

[King of Worms]

wow great job upscaling that face
and the books as well!

[MasonFace]

I've upscaled the different frames from anim0002 and 0004, as well as various illustrations in the books from 0006-0010 and 0014-0015 using a different model

Wow, I really like those results! What model is that, btw?

Here's a quick attempt I made at editing a couple of the different frames together for more detail in the long shots

I'm guessing that you used the detailed face from the upscaled close-up shot to add detail back to the far away shot? The results are fantastic!

[MasonFace]

Well, I got a chance to try out your idea on ANIM0002 and ANIM0004 and I think they turned out quite well!

I used your close up shot upscales and used them to touch up the panned out shots then strung them together in a video.

The transitions are a little too sharp, but I can tweak it if it bothers anyone... the whole video is so short that it's hard to get a good transition in such a short time.

ANIM0002

ANIM0004

[phredreeke]

Here are some edits I made http://s000.tinyupload.com/index.php?fi ... 1255752870

I'll look into reprocessing the upscaled clips later. I may try increasing the framerate as well

Edit: Here's ANIM0003 http://s000.tinyupload.com/index.php?fi ... 6250059006

The image appears to "leave a trail" during the camera movements.

[MasonFace]

Here are some edits I made http://s000.tinyupload.com/index.php?fi ... 1255752870

I've already got ANIM002 and ANIM004 done, see post above. I may have edited it between you first reading it and replying.

I'll look into reprocessing the upscaled clips later. I may try increasing the framerate as well

Edit: Here's ANIM0003 http://s000.tinyupload.com/index.php?fi ... 6250059006

The image appears to "leave a trail" during the camera movements.

I don't think the "trail" looks too bad. I wonder if adding a little bit of random noise either before or after the upscale would help conceal that?


With DAG2, ANIM002, ANIM004 done all we really lack are:

• ANIM003
• ANIM005
• ANIM006
• ANIM007
• ANIM008
• ANIM009
• ANIM010
• ANIM011*
• ANIM012
• ANIM013
• ANIM014
• ANIM015

*ANIM0011 would be the easiest to "upscale" by just creating a new video with some text on it. I think I will try that.

I think all the rest look fine as you've upscaled them, but feel free to keep tweaking them if you want. I've used Butterflow to increase the framerate from 15fps to 60fps after upscaling, but I've had mixed results and I haven't used it in so long that I'd have to re-learn a bit before I could do anything.

The only issue that I see so far that I think really needs to be fixed is the raven animation in ANIM012. If you would separate out the raven frames, I can touch them up and re-animate that short section then stitch it back into the final upscale. Of course if you're still tweaking the upscales then I can wait until you're satisfied.

EDIT: Here is ANIM011

[phredreeke]

alright, did renders of 03,05,12 and 13 with the new script

https://www.dropbox.com/sh/1056i09ys0cx ... 4Ty0a?dl=0

(please download ASAP as I may have to delete it to make space)

Something to consider 1) their final resolution is 1440x1080 but most of the script runs at a lower resolution (1280 wide, height depending on the size of the picture area) since I found that to give better result when uploading to youtube but since the video runs in-game it shouldn't matter much, so what would the desired target resolution be? 2) while the latest script crops the top and bottom to reduce processing times it adds them back near the end to get a 4:3 ratio, should it instead be kept at whatever ratio the image area is?

for the book videos, I think the best course of action would be to try to remake them. the illustrations turns out alright IMO but the text looks pretty ugly (the standalone frames I uploaded are using the fatality model)

a final thing to consider, maybe for certain static shots (the start of anim0012 for example) touch up parts and drop them into the upscale (like the tomb/temple on the right side of the screen)

I'll provide frames for the raven tomorrow.

[MasonFace]

(please download ASAP as I may have to delete it to make space)

I have them downloaded now and added to my collection.

. . . what would the desired target resolution be?

I've been targeting 1080. I don't know that it's the most appropriate, but it looks fine to me.

. . . should it instead be kept at whatever ratio the image area is?

I'm tempted to say that the letterbox should be kept since the original videos were designed for CRT monitors and most people use wide screen formats nowadays.

for the book videos, I think the best course of action would be to try to remake them. the illustrations turns out alright IMO but the text looks pretty ugly (the standalone frames I uploaded are using the fatality model)

I think the upscaled videos you made look great. I don't think its important for the books to have legible text. The important stuff is narrated anyhow.

a final thing to consider, maybe for certain static shots (the start of anim0012 for example) touch up parts and drop them into the upscale (like the tomb/temple on the right side of the screen)

That's a great idea! I might try that.

[Etherian]

I may too late for this suggestion to be useful, but have you looked into StyleGAN? It's a relatively new GAN by Nvidia that has shown effectiveness in a variety of areas. Someone even used it in conjunction with a pipeline similar to yours to create a high-res version of Doomguy's face. It's best known for its amazingly realistic portraits, but seems to perform well on just about any style. Anime, for example. The person who made that anime model wrote up a thorough guide to using StyleGAN if you want to give it a go.

[MasonFace]

I'm aware of StyleGAN and it is very impressive, but it is not strictly an upscaler. The main problem with it is training. It is VERY computationally expensive to train a new model so upscaling vanilla Daggerfall images into high resolution versions would be very difficult. The only models I see that are provided are for generating (not upscaling) faces, cats, cars, and bedrooms.

So, anyway StyleGAN would be great if someone with a powerful GPU farm would train it on already upscaled Daggerfall textures to generate new ones for mods, but to my knowledge, StyleGAN does not operate as an upscaler. Although I suppose the fella who did the Doom guy upscale found a way to use it as such somehow. But again, there are pre-trained models for faces, but not much else at this point.

From what I read, it sounds like the Doom guy upscale took several passes with various AI upscalers and manual artistic touching up before being fed into StyleGAN. There are thousands of textures, portraits, MOBs, UI elements, etc., in the Daggerfall game files. We must have something that is batch-able and requires little-to-no touch up work from an artist or else it will never get completed.



But I really do appreciate the input, and feel free to correct me if I'm wrong.

[MrFlibble]

So after a somewhat lengthy break I resumed my experiments (thanks to Phredreeke who intrigued me with the info about the Fatality model).

I checked out the models listed here and more or less liked the results produced by Rebout and Fatality. The main thing is that you no longer need any pre-processing of the source images, no xBRZ softening or anything of that kind. Also they don't produce JPEG-like artifacts because they were trained on lossless compression data.

I interpolated Fatality and Rebout at alpha 0.5 for what seems like better results:

Code: Select all

net_PSNR_path = './models/rebout_interp.pth'
net_ESRGAN_path = './models/4x_Fatality_01_265000_G.pth'

Here's a sample (the source image was not processed in any way):

I then further interpolated this resulting model (again, at 0.5) with DeToon, which is a model aimed at making images more realistic. You may notice that the entire image became softer and without pronounced black outlines:

It works okay on hand-drawn character flats but not spectacular with the faces.

No DeToon:



With DeToon:


DeToon seems to work better on medium flats and worse on small flats.