AI Upscaled Textures
- MasonFace
- Posts: 543
- Joined: Tue Nov 27, 2018 7:28 pm
- Location: Tennessee, USA
- Contact:
AI Upscaled Textures
UPDATE:
I'm going to try to keep this first post updated with the latest on this project.
---------------------------------------------------------------------
UPDATE:
Satsuki has created a great GUI for ESRGAN. It can even handle pre-filtering! ESRGAN Resizer
Summary:
We are trying to upscale the original Daggerfall textures using a variety of artificial intelligence techniques and some combination of image filters.
All textures have already been upscaled by Freak2121 using Topaz A.I. Gigapixel.
You can download that here: Freak2121's Texture Pack (Gigapixel 4x)
MrFlibble and I have been using various models and pre-processing techniques to enhance upscales using ESRGAN and SFTGAN.
I have created further enhanced textures for Privateer's Hold and created advanced materials for them. You can download this texture replacement mini-mod here (screenshot below):
Masonface's Upscaled Texture Pack (4x + PBR materials) - Privateer's Hold
Note: Make sure filtering mode is set to bilinear or trilinear, not point!
Screenshot: Note: All the textures and more have been rolled into the PVE mod.
---------------------------------------------------------------------
Methodology:
As of now, Freak2121 is getting great results out-of-the-box on textures using Topaz's A.I. Gigapixel.
Other techniques that we've employed are ESRGAN and SFTGAN with mixed results depending on the source image, the model used, and any pre or post processing filters used. These techniques could prove to be the best route if we can train the model on more appropriate data so that it produces results that are higher definition, yet retain the correct aesthetic.
My opinion is that all generated textures/sprites that have been contributed should be made public to the community so anyone can curate their own collection.
The idea there-after is to compile an official collection of textures from all the fore-mentioned techniques, then curate a singular collection that is agreed to best fit for convenience to end-users.
---------------------------------------------------------------------
Original Post:
By suggestion of Luzur, I started tinkering around with some experimental artificial intelligence upscaling methods.
I just started with some basic tests using ESRGAN (https://github.com/xinntao/ESRGAN), and while they aren't perfect, they really ain't half bad! Take a look:
https://imgur.com/zHqCYr0
https://imgur.com/QD83bTw
I think this would be a good way to get a high quality pack of textures rolled out quickly while talented artists take their time remastering the originals (which I think most of us would prefer).
Next is a test using SFTGAN (https://github.com/xinntao/SFTGAN) which I think looks a bit better for textures:
https://imgur.com/YsyXfEe
I'm going to try to keep this first post updated with the latest on this project.
---------------------------------------------------------------------
UPDATE:
Satsuki has created a great GUI for ESRGAN. It can even handle pre-filtering! ESRGAN Resizer
Summary:
We are trying to upscale the original Daggerfall textures using a variety of artificial intelligence techniques and some combination of image filters.
All textures have already been upscaled by Freak2121 using Topaz A.I. Gigapixel.
You can download that here: Freak2121's Texture Pack (Gigapixel 4x)
MrFlibble and I have been using various models and pre-processing techniques to enhance upscales using ESRGAN and SFTGAN.
I have created further enhanced textures for Privateer's Hold and created advanced materials for them. You can download this texture replacement mini-mod here (screenshot below):
Masonface's Upscaled Texture Pack (4x + PBR materials) - Privateer's Hold
Note: Make sure filtering mode is set to bilinear or trilinear, not point!
Screenshot: Note: All the textures and more have been rolled into the PVE mod.
---------------------------------------------------------------------
Methodology:
As of now, Freak2121 is getting great results out-of-the-box on textures using Topaz's A.I. Gigapixel.
Other techniques that we've employed are ESRGAN and SFTGAN with mixed results depending on the source image, the model used, and any pre or post processing filters used. These techniques could prove to be the best route if we can train the model on more appropriate data so that it produces results that are higher definition, yet retain the correct aesthetic.
My opinion is that all generated textures/sprites that have been contributed should be made public to the community so anyone can curate their own collection.
The idea there-after is to compile an official collection of textures from all the fore-mentioned techniques, then curate a singular collection that is agreed to best fit for convenience to end-users.
---------------------------------------------------------------------
Original Post:
By suggestion of Luzur, I started tinkering around with some experimental artificial intelligence upscaling methods.
I just started with some basic tests using ESRGAN (https://github.com/xinntao/ESRGAN), and while they aren't perfect, they really ain't half bad! Take a look:
https://imgur.com/zHqCYr0
https://imgur.com/QD83bTw
I think this would be a good way to get a high quality pack of textures rolled out quickly while talented artists take their time remastering the originals (which I think most of us would prefer).
Next is a test using SFTGAN (https://github.com/xinntao/SFTGAN) which I think looks a bit better for textures:
https://imgur.com/YsyXfEe
Last edited by MasonFace on Thu Jan 23, 2020 1:46 pm, edited 9 times in total.
- Interkarma
- Posts: 7247
- Joined: Sun Mar 22, 2015 1:51 am
Re: AI Upscaled Textures
Thanks for looking into that and sharing results on DF textures. Very fascinating stuff.
I'm amazed how ESRGAN adds those scratches and chips to the left and right sides of window arch, despite those features only being hinted at in original texture. It does take away some of the hand-painted feel in the pixel textures though. But you're right, it's not half bad either.
The SFTGAN does feel a lot more like the original in terms of hand-painted feel to me.
Both of them have trouble with the roughly sculpted arch in original image, but that's more an issue with fidelity of original than anything. Considering the original arched window texture is only 64x128 pixels, very cool!
I'm amazed how ESRGAN adds those scratches and chips to the left and right sides of window arch, despite those features only being hinted at in original texture. It does take away some of the hand-painted feel in the pixel textures though. But you're right, it's not half bad either.
The SFTGAN does feel a lot more like the original in terms of hand-painted feel to me.
Both of them have trouble with the roughly sculpted arch in original image, but that's more an issue with fidelity of original than anything. Considering the original arched window texture is only 64x128 pixels, very cool!
- Hazelnut
- Posts: 3016
- Joined: Sat Aug 26, 2017 2:46 pm
- Contact:
Re: AI Upscaled Textures
Is using these tools at runtime feasible do you think?
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods
-
- Posts: 8
- Joined: Fri Jul 06, 2018 5:10 am
Re: AI Upscaled Textures
I'm far from an expert on graphics, but couldn't human artists tweak the images produced by the AI program instead of starting with the originals? I would imagine it would save them quite a lot of time and effort, yet still give a more handmade look.
- VMblast
- Posts: 520
- Joined: Wed Mar 29, 2017 12:22 pm
- Contact:
Re: AI Upscaled Textures
As an artist I can only say that this is wonderful!
Someone would say to that -what?! But, for a long answer -while upscale like these to someone could look strange, somewhat skewed and far from realistic (painted) look, to me these are wonderful works of impressionism! Now, the classic (human) painting can never be replaced as only artist can do precise work of something looking like beam, window, floor tile, grass etc. But this is really good and interesting stylized alternative. I would very much like to see all the textures redone by these.![Smile :)](./images/smilies/icon_e_smile.gif)
PS - While both systems are great, Im leaning more to SFTGAN version, as it gives better (oil) painted look (you can almost see the brush strokes). Dont get me wrong, ESRGAN is awesome as well, as it is sharper and more illustrative (sketchy -maybe just a tad too contrasty). It is hard to choose between these two, so I would say mixed application would be the best way.
Im looking forward to this experiment as a MOD!
![Smile :)](./images/smilies/icon_e_smile.gif)
Someone would say to that -what?! But, for a long answer -while upscale like these to someone could look strange, somewhat skewed and far from realistic (painted) look, to me these are wonderful works of impressionism! Now, the classic (human) painting can never be replaced as only artist can do precise work of something looking like beam, window, floor tile, grass etc. But this is really good and interesting stylized alternative. I would very much like to see all the textures redone by these.
![Smile :)](./images/smilies/icon_e_smile.gif)
PS - While both systems are great, Im leaning more to SFTGAN version, as it gives better (oil) painted look (you can almost see the brush strokes). Dont get me wrong, ESRGAN is awesome as well, as it is sharper and more illustrative (sketchy -maybe just a tad too contrasty). It is hard to choose between these two, so I would say mixed application would be the best way.
Im looking forward to this experiment as a MOD!
- pango
- Posts: 3359
- Joined: Wed Jul 18, 2018 6:14 pm
- Location: France
- Contact:
Re: AI Upscaled Textures
It may matter to mention not only the algorithms used, but also the models used, as the results depend on the models too (indirectly, on the examples that were learned to produce the model).
Have you done some tests on NPCs/monsters too?
Have you done some tests on NPCs/monsters too?
Mastodon: @pango@fosstodon.org
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart
- King of Worms
- Posts: 4753
- Joined: Mon Oct 17, 2016 11:18 pm
- Location: Scourg Barrow (CZ)
- Contact:
Re: AI Upscaled Textures
Looks interesting, the later example looks better to me.
Is there any download which includes the UI?
Id like to experiment with it, but I dont have any idea how to use the files from the git you linked.
Thank you
Is there any download which includes the UI?
Id like to experiment with it, but I dont have any idea how to use the files from the git you linked.
Thank you
DREAM Nexus https://www.nexusmods.com/users/73250038?tab=user+files
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
-
- Posts: 38
- Joined: Fri Sep 15, 2017 4:01 pm
Re: AI Upscaled Textures
there is a similar experiment going on with doom sprites. However, they're using proprietary tools instead of open source ones. They're also doing a lot of work after the fact, specifically, layered one algorithm's result on top of another. They're also upscaling a LOT and then manually scaling it back down. I think they're upscaling like 8x and then scaling it down so that the final result is 2x larger than the original. It sort of filters out t he weird AI artifacts that way. Might be worth trying some of those strategies.
- MasonFace
- Posts: 543
- Joined: Tue Nov 27, 2018 7:28 pm
- Location: Tennessee, USA
- Contact:
Re: AI Upscaled Textures
Unfortunately, no. Running the upscaling algorithm doesn't take very long to generate each result, but it requires having CUDA installed and having a relatively modern Nvidia graphics card. Also, I think it would take too long to generate ALL the textures needed at run-time.Is using these tools at runtime feasible do you think?
Yes, it could save some time depending on the particular artist's workflow.I'm far from an expert on graphics, but couldn't human artists tweak the images produced by the AI program instead of starting with the originals? I would imagine it would save them quite a lot of time and effort, yet still give a more handmade look.
I think based on the way these two models were trained, they will have differing pros and cons depending on the input textures. For example, it looks like ESRGAN is interpreting the dark regions of the marble to be cracks, where the SFTGAN seems to have a better grasp of what marble is and looks like while also adding in more subtle surface wear to the columns. On stone and earth textures, ESRGAN may fair better. I'll perform a larger sample set and see how they compare in various types of textures, but the SFTGAN workflow is much more tedious and takes longer to get the input textures ready. For ESRGAN, I just have to put the original textures in a folder, then run a single python script and it handles the upscaling. For SFTGAN, I have to first scale the image up to its final size, then run a python script, then run another python script before it begins the upscaling.As an artist I can only say that this is wonderful!![]()
Someone would say to that -what?! But, for a long answer -while upscale like these to someone could look strange, somewhat skewed and far from realistic (painted) look, to me these are wonderful works of impressionism! Now, the classic (human) painting can never be replaced as only artist can do precise work of something looking like beam, window, floor tile, grass etc. But this is really good and interesting stylized alternative. I would very much like to see all the textures redone by these.
PS - While both systems are great, Im leaning more to SFTGAN version, as it gives better (oil) painted look (you can almost see the brush strokes). Dont get me wrong, ESRGAN is awesome as well, as it is sharper and more illustrative (sketchy -maybe just a tad too contrasty). It is hard to choose between these two, so I would say mixed application would be the best way.
From what I read, the SFTGAN that was used in the referenced Git was trained with lots of outdoor photos and that's where it excels. I think that makes it a better fit for most Daggerfall textures overall, but it may not always be the case. Of course, if we had a LARGE collection of textures that have already been hand upscaled by artists, we could train the AI to be better at upscaling the remaining low resolution Daggerfall textures... but I think we would need a LOT of upscaled samples (like maybe 1,000 or more).It may matter to mention not only the algorithms used, but also the models used, as the results depend on the models too (indirectly, on the examples that were learned to produce the model).
I only tried one NPC using the ESRGAN, and the result was really bad.Have you done some tests on NPCs/monsters too?
Yeah, I like the SFTGAN better too on these examples.Looks interesting, the later example looks better to me.
Is there any download which includes the UI?
Id like to experiment with it, but I dont have any idea how to use the files from the git you linked.
There isn't a UI that makes any of this easy; I had to jump through a lot of hoops to get it to run. You need to be fairly comfortable with using terminal commands and you need to have a modern Nvidia graphics card, otherwise you are limited to running the AI (tensor type calculations) on the CPU which is extremely slow.
If you want to give it a go, follow the instructions here: https://www.resetera.com/threads/ai-neu ... elf.88272/
Let me know if you need any help and I'll assist you as best I can (at least if you're using Windows 10).
Yeah, I think they're using the official Nvidia tool which is in closed beta. It would be nice if we had access to it...there is a similar experiment going on with doom sprites. However, they're using proprietary tools instead of open source ones. They're also doing a lot of work after the fact, specifically, layered one algorithm's result on top of another. They're also upscaling a LOT and then manually scaling it back down. I think they're upscaling like 8x and then scaling it down so that the final result is 2x larger than the original. It sort of filters out t he weird AI artifacts that way. Might be worth trying some of those strategies.
![Crying or Very Sad :cry:](./images/smilies/icon_cry.gif)
I had considered layering the two upscaled results on top of each other and maybe using a soft erase brush on the worse looking layer to let the better result show through at that region, but I hadn't considered upscaling to 8x then back down. That's an interesting technique. Thanks for bringing that up!
- King of Worms
- Posts: 4753
- Joined: Mon Oct 17, 2016 11:18 pm
- Location: Scourg Barrow (CZ)
- Contact:
Re: AI Upscaled Textures
In my remaster, I utilized the BRZ4x in two passes, so that means 16x upscale and than manual downscale... so thats a similar aproach. Interresting idea about stacking two algo results in one sprite btw
BTW I applyed for that closed beta nvidia program, and they did not like me I guess, no reply in months
This AI upscaling will be a future of DFU one day, it can produce INSANE results... I saw those at the link u posted, damn!
DREAM Nexus https://www.nexusmods.com/users/73250038?tab=user+files
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz
DREAM Forums viewtopic.php?f=27&t=1168
DISCORD: king_of_worms_cz