Reload/Refresh Textures

Post here if you need help getting started with Daggerfall Unity or just want to clarify a potential bug. Questions about playing or modding classic Daggerfall should be posted to Community.
User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Reload/Refresh Textures

Post by King of Worms »

Can we implement the console command line "reload_tex" to force the engine to reload all the textures from streaming assets?

It would save a lot of time for ppl who do the texture mods - it would allow us to change the textures while in the game, saving us the time we spend at the game restarts after any change we make to textures to be able to see them. It would actually start to resemble the work in a actual game editor. Now its very time costly and ineffective.

Thanks for considering it!

PS: "Clear_cache" to empty the memory could be useful, if thats even possible.
PSS: Is there any way to tell which textures are really used ingame? There are unused texture sets, would be great to know all the unused textures.

User avatar
Igor
Posts: 32
Joined: Sun Nov 19, 2017 10:06 am

Re: Reload/Refresh Textures

Post by Igor »

Already have ideas on what to do with textures? All the textures in memory are not good. I do not have enough memory and everything is very lagging.

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: Reload/Refresh Textures

Post by King of Worms »

Hi, this is beyond me. I hope ppl who can do something about it will do so. We - the moders - are somehow ahead of the schedule. The engine needs to catch up...

For me, what I need is:

- the flats super-atlas to be split into a regular atlases
- animated NPCs handling needs to be changed, it destroys the PC Memory way too much when upscaled
- I was told the NPCs atlases work differently than item flats atlases, I need the specs
+ the things Ive said in the opening post here...

I hope this can be solved, maybe with a cache settings etc. I have absolutely no idea now...

PS: My 1gb upscaled flats pack can saturate and crash a 24gb RAM PC... thats a problem... and it makes no sense to me. Im sure this can be fixed. I wait for Interkarma to be more active again, maybe he can tell us more.

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

King of Worms wrote:Can we implement the console command line "reload_tex" to force the engine to reload all the textures from streaming assets?

It would save a lot of time for ppl who do the texture mods - it would allow us to change the textures while in the game, saving us the time we spend at the game restarts after any change we make to textures to be able to see them. It would actually start to resemble the work in a actual game editor. Now its very time costly and ineffective.

Thanks for considering it!

PS: "Clear_cache" to empty the memory could be useful, if thats even possible.
Let me see what I can do. The ability flush texture and model caches and reset scene are already in the code, would just need to wire it up to console commands. Are you happy for this to trigger a save/load automatically after executing command? That's the most direct way to fully purge and reload a scene.
King of Worms wrote: PSS: Is there any way to tell which textures are really used ingame? There are unused texture sets, would be great to know all the unused textures.
I can't provide a definitive list of unused textures, I'm sorry. :(

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

Igor wrote:Already have ideas on what to do with textures? All the textures in memory are not good. I do not have enough memory and everything is very lagging.
The textures are only loaded when needed, with a few key exceptions (like terrain sets) that need to be prepared ahead of time for the world to stream. The cache does store textures after they're loaded however, which can lead to bloat over time. A normal play session (without mods) should only consume around 1-2GB last time I checked. Naturally, huge texture mods will bloat memory relative to the size of new textures injected.
King of Worms wrote:Hi, this is beyond me. I hope ppl who can do something about it will do so. We - the moders - are somehow ahead of the schedule. The engine needs to catch up...

For me, what I need is:

- the flats super-atlas to be split into a regular atlases
- animated NPCs handling needs to be changed, it destroys the PC Memory way too much when upscaled
- I was told the NPCs atlases work differently than item flats atlases, I need the specs
+ the things Ive said in the opening post here...

I hope this can be solved, maybe with a cache settings etc. I have absolutely no idea now...

PS: My 1gb upscaled flats pack can saturate and crash a 24gb RAM PC... thats a problem... and it makes no sense to me. Im sure this can be fixed. I wait for Interkarma to be more active again, maybe he can tell us more.
Something to keep in mind is that most really huge textures in modern games will use GPU compression (e.g. DXT4). My texture manager supports texture compression, but it's turned off by default as it wrecks the clean pixel look of vanilla Daggerfall's low-fi sprites. It's not much of a problem for large textures though.

I don't believe texture compression is exposed to the end user in INI settings, and I'm not sure if TheLacus' texture injection will set the compression format for injected textures either. But exposing this would be a great place to start and probably fix a majority of memory problems.

The next step is likely to disable atlasing when injecting large textures. The atlas is a great way to store lots of small textures and save on state changes, but it doesn't scale up very far once you start throwing around individual 2k or 4k textures.

You've probably noticed that TheLacus is driving most of the mod support right now (and doing an amazing job of it!). I'm more focused on the gameplay features exclusively. Unfortunately some of the choices I made early on for optimisation were made specifically for Unity 4.x Free (back when Free was heavily cut down). And I didn't really foresee how rapidly the mod scene would grow either. You guys are like a year ahead of where I thought you'd be by now haha. I'm sure things will keep improving slowly over time. :)

User avatar
Igor
Posts: 32
Joined: Sun Nov 19, 2017 10:06 am

Re: Reload/Refresh Textures

Post by Igor »

Interkarma wrote: The textures are only loaded when needed, with a few key exceptions (like terrain sets) that need to be prepared ahead of time for the world to stream. The cache does store textures after they're loaded however, which can lead to bloat over time. A normal play session (without mods) should only consume around 1-2GB last time I checked. Naturally, huge texture mods will bloat memory relative to the size of new textures injected.
In the field of view a maximum 256 megabytes of textures. But the game consumes 9-10 gigabytes of RAM. I only use the Grafon and no other mods.

User avatar
TheLacus
Posts: 1305
Joined: Wed Sep 14, 2016 6:22 pm

Re: Reload/Refresh Textures

Post by TheLacus »

Interkarma wrote: The textures are only loaded when needed, with a few key exceptions (like terrain sets) that need to be prepared ahead of time for the world to stream. The cache does store textures after they're loaded however, which can lead to bloat over time. A normal play session (without mods) should only consume around 1-2GB last time I checked. Naturally, huge texture mods will bloat memory relative to the size of new textures injected.
Mod system also has a dictionary for each mod so assets are loaded from assetbundle only once. I can see how imported textures and models can start to take up a lot of memory, especially if you travel to differents climates and have different variations where vanilla only swapped texture on mesh.
Something to keep in mind is that most really huge textures in modern games will use GPU compression (e.g. DXT4). My texture manager supports texture compression, but it's turned off by default as it wrecks the clean pixel look of vanilla Daggerfall's low-fi sprites. It's not much of a problem for large textures though.

I don't believe texture compression is exposed to the end user in INI settings, and I'm not sure if TheLacus' texture injection will set the compression format for injected textures either. But exposing this would be a great place to start and probably fix a majority of memory problems.
Image data imported from png without the editor can be loaded into RGBA32 or DXT5, so it should be straightforward to add this option. Which existing field should the ini key be linked with, if any?

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

TheLacus wrote: Image data imported from png without the editor can be loaded into RGBA32 or DXT5, so it should be straightforward to add this option. Which existing field should the ini key be linked with, if any?
MaterialReader.CompressSkyTextures used to be a field that applied to all textures (I think it was just called "CompressTextures" at the time). The setting had a negative impact on small textures such as flats, so I just retained setting for sky images where compression was barely noticeable. The same should be true for any large texture.

I'd still rather not have texture compression for the classic textures, as it corrupts their aesthetic and has minimal impact on memory usage. Perhaps we can add a new MaterialReader field called "CompressModdedTextures" or similar, and load as compressed when field is true. It could even be enabled by default.

Or would it be better to make this a setting in the mod manager so it can be toggled on a per-mod basis?

User avatar
Interkarma
Posts: 7234
Joined: Sun Mar 22, 2015 1:51 am

Re: Reload/Refresh Textures

Post by Interkarma »

Igor wrote: In the field of view a maximum 256 megabytes of textures. But the game consumes 9-10 gigabytes of RAM. I only use the Grafon and no other mods.
As I said, unmodded Daggerfall Unity only consumes around 1-2GB of RAM. Large texture mods will require a lot more memory as nothing is compressed at this time. If a setting is added to compress textures, the ratio should be around 4:1.

Couple this with some cache expiry behaviour and RAM requirements for large textures mods should reduce substantially. I imagine this will get better over time as mod support matures. You guys are way ahead of the curve here. :)

Even with these changes, modders should observe rational constraints. Throwing insanely huge textures into a game engineered around a world with average textures of 64x128 pixels is obviously not going to end well. Consider user experience in addition to appearance value, and find the best balance between both.

User avatar
King of Worms
Posts: 4747
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow (CZ)
Contact:

Re: Reload/Refresh Textures

Post by King of Worms »

1st, very glad to see you back Interkarma! :)

- "Are you happy for this (reset scene) to trigger a save/load automatically after executing command? That's the most direct way to fully purge and reload a scene."
:arrow: That would be fantastic, this whole option would save me so much time and as Ive said, with this, the work on textures would really resemble work in a game engine editor where u can see the changes in real time.


- "I can't provide a definitive list of unused textures, I'm sorry"
:arrow: no problem, I guess the textures which are not used ingame, are not loaded/cached anyway, so its not a performance problem if we have the unused textures in high resolution. I have already a very good idea on unused textures anyway.

- "...disable atlasing"
:arrow: I was able to fit the textures in standard atlasses, those 4k*4k ones. My problem is with the MiscFlats super-atlas which combines archives 97, 205, 211, 212, 213, 301 with size 2048. That super atlas prevents all my upscale efforts in that region. But Im ok with whatever solution to this.

- "...texture compression for high res tex + some cache expiry behaviour (Yes!!) and RAM requirements for large textures mods should reduce substantially."
:arrow: I think if you leave the dungeon, all dungeon textures and enemy NPCs should be wiped from MEM. Plus when you travel, on map from sentinel to daggerfall, than desert textures and flats are not needed anymore etc - it can be made by expiry time as well. Cache is huge now after a while of playing, and that is with my basic 500Mb textures of few buildings etc... 8gb ram, 16gb paging file, 4gb VRAM... all full... Texture compression should be a option tho. My textures are low in size in terms of Mb, but they are quite high resolution (thats the nature of upscaling the vanila textures) I dont think they need compression, but I dont really know.

Textures work quite ok for me, flats/sprites on the other hand - very problematic! Especially animated NPCs + misc superatlas issue.


The problem with upscaled enemy NPCs in dungeons for ex. they just massacre the system memory. Dont know what is going on there. Every upscaled NPC set has like 10-30mb on HDD. Ingame, it goes to 10s of gigabites of memory... I THINK maybe if the dung has 100 enemies, it caches each enemy? So 20*30mb for 20 rats. 10*30mb for 10 skeletons. 20*30mb for 20 bats etc?

Anyway, thanks for responses to you all!

Post Reply