Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Discuss modding questions and implementation details.
sayber1
Posts: 46
Joined: Thu Sep 16, 2021 3:27 pm

Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by sayber1 »

This is a disscusion I've started in a Vanilla Enchanced mod thread. In this topic I want to elaborate why I don't think it's a good pratice to use artificial upscaling and filtering on pixel art, at least at the current state of technology.

The unique defining feature of pixel art that seaparates it from a lot of other forms of digital art, is how incredibly deliberate it is. Because of limited estate every pixel in a pixel artwork is placed deliberately and with a lot of thought. Unlike hi-rez images there you can allow little slips here and there and them being relatively unnoticeable each pixel in a pixel art artwork has a purpuse.

The AI methods available currently do not have a neccecery awareness for properly working with a pixel art. To illustrate this I have to use this filtered texture from carademono's mod
Spoiler!
2022_01_21_23_50_59.jpg
2022_01_21_23_50_59.jpg (286.71 KiB) Viewed 1259 times
2022_01_21_23_50_15.jpg
2022_01_21_23_50_15.jpg (290.18 KiB) Viewed 1259 times
While it can be argued that it improved clarity on the sides of window frames and the twigs, it also completely messed up the shadows under the leaves, lower parts of the iron barn on a window, added discrapency by coloring only upper nails in the plank and subtly messed up the tiling of the texture.

Thats the one of the general problems with current AI techniques. While it makes some things subjectively better, it usualy makes even more things objectively worse.

I would like to touch on the other aspects of this problem in the later posts, but I think this should be suffucient to start discussion.

User avatar
BadLuckBurt
Posts: 948
Joined: Sun Nov 05, 2017 8:30 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by BadLuckBurt »

Just out of curiosity, what are your criteria for defining pixel art?

I'm not a fan of AI upscaling / filtering either but that's not reason enough for other people not to use them. The only way to improve these processes is by using them after all, not by denying them because they're 'not ready' yet.

A computer will never beat a human when it comes to creating art, the decision process differs too much between the two. It can produce a decent enough result when it's been trained well enough. Someone has created neural AI upscales for Doom textures and most of those look pretty damn good, it really depends on the person using the tool and the amount of material available to train the AI.

At the end of the day, there is nothing more subjective than art, it doesn't even have to be beautiful to be considered beautiful. Daggerfall's artstyle is all over the place even without replacements but that doesn't stop anyone from appreciating it for what it is.

Some of the sprite proportions are completely off, the styles clash and there are plenty of 'pixel artifacts' to point out where the palettization of an image introduced some very offensive color choices for certain pixels. This is because most of Daggerfall's texture were most likely not drawn pixel by pixel but instead painted, probably using DeluxePaint or a similar program. This isn't like JRPGs or Metal Slug.

You can find higher resolution versions of Daggerfall textures in Battlespire too and those definitely weren't done pixel by pixel.

Don't even get me started on the quality of some of the models in Daggerfall :lol: I still enjoy the hell out of the game though.

It's fine if your opinion is that this upscaling shouldn't be done, that won't stop others however. Some people just don't like pixels while others love them, there is no right or wrong regardless of what you believe.
DFU on UESP: https://en.uesp.net/w/index.php?title=T ... fall_Unity
DFU Nexus Mods: https://www.nexusmods.com/daggerfallunity
My github repositories with mostly DFU related stuff: https://github.com/BadLuckBurt

.

sayber1
Posts: 46
Joined: Thu Sep 16, 2021 3:27 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by sayber1 »

The pixel art is what I would define an artwork made out of extremely limited amout of pixels and constraint by that amount to point of needing to adapt the style to make it readable enough.

While i agree that art is subjective, there are objective criteria that needs to be taken into consideration. Take color theory for example. Personally I never liked the Joker(purple+green) classic color scheme, but that doesn't make it bad. It completely follows the principles of the color theory and is all in all a good color scheme. It is subjective. But if you take 255:0:0 Red and 0:255:0 Blue and make it a color scheme it would be completly awful as those colors do not work together in an objective sense. The hardest part of critisizing art is seeing this distinction between objective issues and subjective impression.

The issue I see, is the same as with 60fps animation that due to being easy to make, and not being that distracting to people not versed in the art, makes some people believe it to be a genuine improvement and potentially leading to people overrelying on it. As I said I would not want it to become a standart in DFU community.

As for Daggerfall's mess of an artstyle I do agree. It's because of this I am still currently working on redrawing some of it's worst, reused from Arena, NPC flats. But the things that are not messed up are actually pretty good. They may not be at the top level of pixel art, but they are still nice and solid artworks. I do believe that we need a proper fix for those issues, a well thought project that would manually fix the messier parts of DF art.

That said AI filtering I believe would only worsen this situation. As it only adds more mess to existing pixel art. And people in general may tend to ignore this issue more and more, not being able to properly evaluate the problems of that method. It's hard for me to choose the correct words, but basically what I mean is people thinking that the problem is already fixed looking at upscaled version of an artwork and not contributing to a genuine improvement. Thinking that a problem is solved when it is not.

User avatar
BadLuckBurt
Posts: 948
Joined: Sun Nov 05, 2017 8:30 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by BadLuckBurt »

sayber1 wrote: Sat Jan 22, 2022 10:17 am The pixel art is what I would define an artwork made out of extremely limited amout of pixels and constraint by that amount to point of needing to adapt the style to make it readable enough.
So, the King of Fighter hi-res pixel art does not count? https://kofaniv.snk-corp.co.jp/english/ ... /index.php
sayber1 wrote: Sat Jan 22, 2022 10:17 am While i agree that art is subjective, there are objective criteria that needs to be taken into consideration. Take color theory for example. Personally I never liked the Joker(purple+green) classic color scheme, but that doesn't make it bad. It completely follows the principles of the color theory and is all in all a good color scheme. It is subjective. But if you take 255:0:0 Red and 0:255:0 Blue and make it a color scheme it would be completly awful as those colors do not work together in an objective sense. The hardest part of critisizing art is seeing this distinction between objective issues and subjective impression.
While I'm colourblind, the EGA palette is a good example of an offending colour palette. People still made wonderful graphics with it, you might find this video interesting: https://www.youtube.com/watch?v=aMcJ1Jvtef0. Giving extreme examples (the 255 colours are notoriously bright and clash with just about anything) does not help to illustrate your point. If anything it's all about balance which in the case of colours would be the theory you mention.

Objective criteria are guidelines, not absolute rules to be followed. If no-one ever broke the mold, there would be no new art. If there's anything an artist should remember, it's dare to be bold.
sayber1 wrote: Sat Jan 22, 2022 10:17 am The issue I see, is the same as with 60fps animation that due to being easy to make, and not being that distracting to people not versed in the art, makes some people believe it to be a genuine improvement and potentially leading to people overrelying on it. As I said I would not want it to become a standart in DFU community.
You don't get to dictate the standard nor does anyone else. Your opinion of things is as worthless as my own, at the end of the day no-one cares and will do and use whatever they want to have the experience they're looking for and that's as it should be.
sayber1 wrote: Sat Jan 22, 2022 10:17 am As for Daggerfall's mess of an artstyle I do agree. It's because of this I am still currently working on redrawing some of it's worst, reused from Arena, NPC flats. But the things that are not messed up are actually pretty good. They may not be at the top level of pixel art, but they are still nice and solid artworks. I do believe that we need a proper fix for those issues, a well thought project that would manually fix the messier parts of DF art.
I'm happy to hear that, your post in Carademono's thread would've been received in a much different light had you mentioned this instead of saying you don't see the point of such a mod. I'm pretty sure everyone would welcome a mod that corrects the entire artstyle of Daggerfall so the quality is more consistent. You can bet your behind that when you release it, there will be this unavoidable question from people: Can you make it DREAM compatible or are there plans for that?
sayber1 wrote: Sat Jan 22, 2022 10:17 am That said AI filtering I believe would only worsen this situation. As it only adds more mess to existing pixel art. And people in general may tend to ignore this issue more and more, not being able to properly evaluate the problems of that method. It's hard for me to choose the correct words, but basically what I mean is people thinking that the problem is already fixed looking at upscaled version of an artwork and not contributing to a genuine improvement. Thinking that a problem is solved when it is not.
For some people, who don't care about it as much, the AI filtering is the solution. Nothing you do is going to change that, some people just don't like pixels staring them in the face. I grew up on 320x200 resolution but we have screens running 10 times that resolution now so I can definitely see why some people have issues with it and would prefer to go with an upscale. Do they miss out? Yes, I tend to think so but that's no reason to scold them or look down on the work of the people that created those upscales, in the end it's their decision and their loss.
DFU on UESP: https://en.uesp.net/w/index.php?title=T ... fall_Unity
DFU Nexus Mods: https://www.nexusmods.com/daggerfallunity
My github repositories with mostly DFU related stuff: https://github.com/BadLuckBurt

.

sayber1
Posts: 46
Joined: Thu Sep 16, 2021 3:27 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by sayber1 »

BadLuckBurt wrote: Sat Jan 22, 2022 11:15 am
So, the King of Fighter hi-res pixel art does not count?
It's exactly what I was saying tho. While KOF sprites are higher resolution than most other pixel art, they are still using limited palette, limited resolution to a point of having make consessions in artstyle(Look at the solid colors in shadows) and master this.
BadLuckBurt wrote: Sat Jan 22, 2022 11:15 am
While I'm colourblind, the EGA palette is a good example of an offending colour palette. People still made wonderful graphics with it, you might find this video interesting: https://www.youtube.com/watch?v=aMcJ1Jvtef0. Giving extreme examples (the 255 colours are notoriously bright and clash with just about anything) does not help to illustrate your point. If anything it's all about balance which in the case of colours would be the theory you mention.

Objective criteria are guidelines, not absolute rules to be followed. If no-one ever broke the mold, there would be no new art. If there's anything an artist should remember, it's dare to be bold.
I don't quite understand. In this video I see a masterful use and understanding of color theory. EGA palette is pretty decent, considering all thing. Raw CGA on the other hand is a better example of almost irredimable uglyness. Of course with a sufficient knowledge and skills you could use pretty bold colors and create decent thing out of it. But it's precisely about avoiding bad color combinations and masking them as well as you can. What I was trying to say is there are objectively bad color combinations and subjectively bad combinations that you might not like, but that doesn't make them objectively bad. Same with other(mostly technical) aspects of art.
BadLuckBurt wrote: Sat Jan 22, 2022 11:15 am
I'm happy to hear that, your post in Carademono's thread would've been received in a much different light had you mentioned this instead of saying you don't see the point of such a mod. I'm pretty sure everyone would welcome a mod that corrects the entire artstyle of Daggerfall so the quality is more consistent. You can bet your behind that when you release it, there will be this unavoidable question from people: Can you make it DREAM compatible or are there plans for that?
It was late night for me, and I have not been able to indicate my thoughts properly. When I said I don't understand that mod, I actually ment it, as there might have been a bigger picture and a plan that I couldn't see. I as already said I really messed up it that thread and was unitentually really rude.

Sadly I'm not able to dedicate myself enough to such large scale project, having to settle with smaller additions and improvements. So I want to at least give this matter an attention of other people.

As for the last part, I agree. But people have different standarts, and it's only natural to try and protect and expand your view on the situation. And I'm not fighting with DREAM or other mods that try to bring resolution and fidelity to modern standarts, tho I do believe this effort to be futile. We started this disscusion in a topic about the mod that worked on low res enhancement, which did it in a way I believe was not correct.

User avatar
BadLuckBurt
Posts: 948
Joined: Sun Nov 05, 2017 8:30 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by BadLuckBurt »

sayber1 wrote: Sat Jan 22, 2022 11:51 am It's exactly what I was saying tho. While KOF sprites are higher resolution than most other pixel art, they are still using limited palette, limited resolution to a point of having make consessions in artstyle(Look at the solid colors in shadows) and master this.
Daggerfall's palette limitations are a result of the hardware at the time, not a deliberate choice I'd say. The colors themselves were probably deliberately chosen or at least tweaked and it allowed them to do the palette swap for clothes etc. saving precious disk space but had they been given the choice not limited by hardware, I bet they would've used more colours. Also there were originally 16 frames of animation instead of the 8 per character that we got. It's still a far cry from the 60 frames but it still would've been a smoother experience than what we ended up getting.
sayber1 wrote: Sat Jan 22, 2022 11:51 am I don't quite understand. In this video I see a masterful use and understanding of color theory. EGA palette is pretty decent, considering all thing. Raw CGA on the other hand is a better example of almost irredimable uglyness. Of course with a sufficient knowledge and skills you could use pretty bold colors and create decent thing out of it. But it's precisely about avoiding bad color combinations and masking them as well as you can. What I was trying to say is there are objectively bad color combinations and subjectively bad combinations that you might not like, but that doesn't make them objectively bad. Same with other(mostly technical) aspects of art.
Mark Ferrari is one of the masters of this craft but if you watch the whole video, you'll see that Mark wasn't happy with EGA, he explains why and shows examples too. CGA is not pretty, I agree but I've still seen great art made with it. The thing is, objectively bad colors or anything comparable like musical chords can still look / sound right when done right. I think our perspectives on this are just way too different to reach a real consensus between the two of us :)
sayber1 wrote: Sat Jan 22, 2022 11:51 am It was late night for me, and I have not been able to indicate my thoughts properly. When I said I don't understand that mod, I actually ment it, as there might have been a bigger picture and a plan that I couldn't see. I as already said I really messed up it that thread and was unitentually really rude.
Fair enough, I'm not going to bring it up again. We've all been there where we were too tired or unfocused to get our point across.
sayber1 wrote: Sat Jan 22, 2022 11:51 am Sadly I'm not able to dedicate myself enough to such large scale project, having to settle with smaller additions and improvements. So I want to at least give this matter an attention of other people.
Which is fine, it's not an easy task to rectify assets so with some luck others might join the effort when you publish your work. Stranger things have happened :)
sayber1 wrote: Sat Jan 22, 2022 11:51 am As for the last part, I agree. But people have different standarts, and it's only natural to try and protect and expand your view on the situation. And I'm not fighting with DREAM or other mods that try to bring resolution and fidelity to modern standarts, tho I do believe this effort to be futile. We started this disscusion in a topic about the mod that worked on low res enhancement, which did it in a way I believe was not correct.
From my perspective, I don't think there is anything to protect when it comes to modding, especially in this small community. Discussion is always welcome though so I'm glad you made this post.
DFU on UESP: https://en.uesp.net/w/index.php?title=T ... fall_Unity
DFU Nexus Mods: https://www.nexusmods.com/daggerfallunity
My github repositories with mostly DFU related stuff: https://github.com/BadLuckBurt

.

sayber1
Posts: 46
Joined: Thu Sep 16, 2021 3:27 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by sayber1 »

Well thanks. It's okay for us to disagree on those things, and dispite a lot of things I said, I'm still just happy there are people working to improve this game. Even if stuff doesn't fully comform to what I would want. I can just try my best to contribute in the way I find the the right.

User avatar
BadLuckBurt
Posts: 948
Joined: Sun Nov 05, 2017 8:30 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by BadLuckBurt »

As the saying goes, the more the merrier :) We all have our own visions and approaches but to quote a certain Beth employee: 'It just works' :?
DFU on UESP: https://en.uesp.net/w/index.php?title=T ... fall_Unity
DFU Nexus Mods: https://www.nexusmods.com/daggerfallunity
My github repositories with mostly DFU related stuff: https://github.com/BadLuckBurt

.

User avatar
carademono
Posts: 210
Joined: Sat Jul 11, 2020 3:20 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by carademono »

sayber1 wrote: Sat Jan 22, 2022 1:02 pm Well thanks. It's okay for us to disagree on those things, and dispite a lot of things I said, I'm still just happy there are people working to improve this game. Even if stuff doesn't fully comform to what I would want. I can just try my best to contribute in the way I find the the right.
I actually rather agree with a lot of the points you've made, sayber1. Not about my whole approach being wrong, of course, or about AI methods not being suitable for remastering pixel art (I don't see any way to increase the color depth from 8-bit to 32-bit color without them). But I do agree that AI has a tendency to improperly handle many details and undermine the artist's intent in those rare cases where the Daggerfall pixel art was originally created by hand. With Vanilla Enhanced I spent a lot of time going through the textures and manually adjusting the blends between AI and vanilla layers. That task is unfinished, and there are also texture seams I hope to fix over time.

If you genuinely think the whole approach is wrong, then no need to play my mod. But if on second glance you think the approach is viable but could use a lot more manual correction, then I would love for you to contribute to the mod. All the files are available on the following GitHub repositories:
https://github.com/drcarademono/vanilla-enhanced
https://github.com/drcarademono/vanilla-enhanced-skies
https://github.com/drcarademono/villager-variety
https://github.com/drcarademono/villain ... er-variety
https://github.com/drcarademono/animated-people

You can use GitHub Desktop to sync these to your computer, edit textures you think still have issues, and submit those back to the main repo as a pull request. I've already seen some of your excellent work on the 184 archive NPC flats and NPC animations, and would love for you to consider making a VE version of those.

I'd be really happy to see lots of community involvement in these mods, because I do want to see them get continually better and because (with a new baby on the way) pretty soon I won't have as much time to keep developing them so intensively.

sayber1
Posts: 46
Joined: Thu Sep 16, 2021 3:27 pm

Re: Using AI upscalling and filtering techniques on a pixel art and why I think it's not there yet.

Post by sayber1 »

carademono wrote: Sat Jan 22, 2022 6:03 pm
If you genuinely think the whole approach is wrong, then no need to play my mod. But if on second glance you think the approach is viable but could use a lot more manual correction, then I would love for you to contribute to the mod. All the files are available on the following GitHub repositories.
Thanks carademono, as I said I do not think AI enhancement a viable method but I think I will look forward for your work on this project. Perhaps I am wrong and it may be a more viable aproach than I thought. Of course I would provide you with my for for 184 upscale as soon as I properly finish it(due to long break, I'm still, I'd say, only 2/3 of way there).

On a side note may I suggest to tune down the smoothing on some NPC textures of VE as It tends to loose too much details
2022_01_22_01_43_57.jpg
2022_01_22_01_43_57.jpg (296.33 KiB) Viewed 1082 times
2022_01_22_01_42_20.jpg
2022_01_22_01_42_20.jpg (294.09 KiB) Viewed 1082 times
Tho maybe you are already working on this, or these are placeholders.

Post Reply