Reload/Refresh Textures

Post here if you need help getting started with Daggerfall Unity or just want to clarify a potential bug. Questions about playing or modding classic Daggerfall should be posted to Community.
ifkopifko
Posts: 195
Joined: Thu Apr 02, 2015 9:03 am

Re: Reload/Refresh Textures

Post by ifkopifko »

Just a note: PNG files are compressed, it is lossless compression, but it is compressed. For 1024x2048 pixels, 24 bits per pixel, it is 6MB uncompressed. So all your upscaled textures could take up 10GB in 24bit bitmap format. Not sure if it would explain the behaviour you described though.

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

King of Worms wrote: The problem with upscaled enemy NPCs in dungeons for ex. they just massacre the system memory. Dont know what is going on there. Every upscaled NPC set has like 10-30mb on HDD. Ingame, it goes to 10s of gigabites of memory... I THINK maybe if the dung has 100 enemies, it caches each enemy? So 20*30mb for 20 rats. 10*30mb for 10 skeletons. 20*30mb for 20 bats etc?
ifkopifko wrote:Just a note: PNG files are compressed, it is lossless compression, but it is compressed. For 1024x2048 pixels, 24 bits per pixel, it is 6MB uncompressed. So all your upscaled textures could take up 10GB in 24bit bitmap format. Not sure if it would explain the behaviour you described though.
ifkopifko pretty much nails it. The size of source image on disk doesn't correspond with how engine stores that image internally for GPU. Currently this is RGBA32 format, so a full 32-bits per pixel. With DXT5 compression, this will come down to 4:1 ratio, or around 8-bits per pixel on average with almost the same overall fidelity.

I'll see what I can do with improved cache management once modded textures are compressed. :)

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: Reload/Refresh Textures

Post by King of Worms »

This sounds very promising, thanks Interkarma. Many games had "compress textures" option, and usually, I was not able to tell the difference between compressed and not. So lets hope it will work here as well. Fingers crossed. And thanks Pifko for the explanation as well!

PS: Its interresting to me, that the original 256 color textures benefit from the 32 bit depth, but I dont understand jack in this department anyway ;) maybe it adds up to smooth gradients when its rendered

User avatar
TheLacus
Posts: 1305
Joined: Wed Sep 14, 2016 6:22 pm

Re: Reload/Refresh Textures

Post by TheLacus »

Interkarma wrote: MaterialReader.CompressSkyTextures used to be a field that applied to all textures (I think it was just called "CompressTextures" at the time). The setting had a negative impact on small textures such as flats, so I just retained setting for sky images where compression was barely noticeable. The same should be true for any large texture.

I'd still rather not have texture compression for the classic textures, as it corrupts their aesthetic and has minimal impact on memory usage. Perhaps we can add a new MaterialReader field called "CompressModdedTextures" or similar, and load as compressed when field is true. It could even be enabled by default.
Sounds good. The only issue i found is a discrepancy in normal maps: they look different, somewhat lighter, with DXT5 (second picture) compared to ARGB32. I'm aware Unity uses alpha and green channel, but afaik is the same for both formats.
normals (1).png
normals (1).png (1.72 MiB) Viewed 4324 times
normals (2).png
normals (2).png (1.48 MiB) Viewed 4324 times
Or would it be better to make this a setting in the mod manager so it can be toggled on a per-mod basis?
Do you think we should call Compress on textures imported with models? Compression is enabled by default in editor so many mods will probably leave it as is.

Textures for vanilla meshes, npcs, etc. are all imported from StreamingAssets so only a global toggle is possible.

I have been thinking about importing materials from mods because this would fit great with load order (if mod0 has albedo + normal and mod1 only albedo, only the latter is imported, while dropping everything on disk can create unintended mixes) but also has cons (unity editor is required, atlases and arrays requires textures to be stripped, mods like Nystul reflections inject only specfic properties like metallicgloss on purpose) so i kept this discussion for more mature times.

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

TheLacus wrote: Sounds good. The only issue i found is a discrepancy in normal maps: they look different, somewhat lighter, with DXT5 (second picture) compared to ARGB32. I'm aware Unity uses alpha and green channel, but afaik is the same for both formats.
Interesting. Not sure what is causing that right now. Can any other Unity devs weigh in?

TheLacus wrote: Do you think we should call Compress on textures imported with models? Compression is enabled by default in editor so many mods will probably leave it as is.

Textures for vanilla meshes, npcs, etc. are all imported from StreamingAssets so only a global toggle is possible.
Global is fine for loose textures, and mod creators have control over this themselves as they're packaging with editor. If there are any problems with consistency, this can be revisited later.

TheLacus wrote: I have been thinking about importing materials from mods because this would fit great with load order (if mod0 has albedo + normal and mod1 only albedo, only the latter is imported, while dropping everything on disk can create unintended mixes) but also has cons (unity editor is required, atlases and arrays requires textures to be stripped, mods like Nystul reflections inject only specfic properties like metallicgloss on purpose) so i kept this discussion for more mature times.
Yeah, this is a tougher one. I think the loose textures on disk are great for testing. But anyone distributing a large-scale mod should take the time to create a .dfmod package for user convenience. But this does complicate the pipeline in other ways as you say. Agreed it's best to defer for now until the systems are a bit more mature.

User avatar
Midknightprince
Posts: 1324
Joined: Fri Aug 11, 2017 6:51 am
Location: San Antonio TX
Contact:

Re: Reload/Refresh Textures

Post by Midknightprince »

The memory thing only happend to me once.
Only in one particular dungeon, I took the monsters and stuff out after that.
The game seems to run at about 2 to 3 gigs with all the mods and textures working, even before I took them out, so they seemed to be working.
The loading time was a bit long but the game didn't lag really at all, then I go into a dungeon, and crash..
It seems like it's doing something weird, and the texture atlas thing wasn't messing up. that's all I know....
Hope this helps !
Check out my YouTube Channel!

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: Reload/Refresh Textures

Post by King of Worms »

With all the respect Midnightprince, your post has no ground.

"texture atlas thing wasn't messing up" - you have no idea how misc super atlas overflow manifests. I remember you posting a screenshot in my thread with a bugged ceiling lamp, and that bug was caused by atlas overflow, and the lamp is from one of the misc texture sets merged in the super atlas so please...

Ive spend 100s of hours on this, monitoring the engine behavior, memory usage, paging file usage, vram usage... Do you really need to come here and tell me the issues I face and describe here and around all the forums for a month are not a problem? And that is supposed to help with what?

"The game seems to run at about 2 to 3 gigs with all the mods and textures working" - seems? 2 - 3 gigs of what? RAM, VRAM, PAGING FILE? What do you mean by "all the mods" anyway?

I understand well you like to write and post and I appreciated it the whole time, but this is not the place and this is not the way right here.

Btw, I know you have 24gb ram, so maybe that is the cause why "memory thing" happened to you only once? I remember you have not even used MSI Afterburner, when I wanted exact figures from you...

Who knows because you described your setup like this:

"24 gigs@1333 I think ?
GTX 1050TI OC 4 gigs.
Total vid mem is 16+ gigs."

...yes, makes sense.

Listen, Im not here to bash you, but you dont seem to be on the tech side. Im not either, but I dont come to your threads, where you try to describe the problems you facing when you create mods to the game, and I dont tell you and others these issues are not a problem. Im afraid you do not realize some things right now.

To me, this is upsetting and also insulting, I cant help it, sorry!

Here is YOUR OWN PICTURE capturing ATLAS OVERFLOW - the thing you say doesnt exists now all of sudden while using the same build 92 - and I will end it here
Light.jpg
Light.jpg (523.59 KiB) Viewed 4234 times

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

KoW - what are your thoughts on us disabling the super atlas for misc flats? My rationale for this atlas in the first place was these flats were all very tiny and could be easily combined to reduce draw calls, but this obviously makes it harder to scale up flat sizes in mods. That's entirely my lack of foresight.

Even if each misc flat archive was confined to its own atlas (like other regular flats), it would make these overflow problems much less of an issue.

What do you reckon? Thumbs down to super atlas? :)

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: Reload/Refresh Textures

Post by King of Worms »

Hi Interkarma!

Separating this super atlas so "each misc flat archive was confined to its own atlas" would solve a lot of problems with sprite/flats modding. Death to the super atlas :twisted: - eventho its a great idea and I understand where it came from from a optimalization point fo view.

Btw, if The Lacus is right - and he always is ;) - than he said:
"97, 205, 211, 212, 213, 301 are in the same "MiscFlats super-atlas with size 2048"

and my reply to that - after some tinkering in the archives - is:

"097 - 22x npc size statues
205 - 45x container
211 - 58x misc items
212 - 20x outdoor items, some major
213 - 18x plants, some major
301 - 24x crops
--------------------------------------------------
187 Objects, many of them big in size and important. - and all are finished on my HDD
This CANT be in one atlas"



I would be very happy if this can be done, if thats not a big problem. Thanks a lot!

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

Agreed! Death to the super atlas it is. :)

I'll try to get this changed in next day or so.

Post Reply