Texture conversion and caching question
- numidium3rd
- Posts: 188
- Joined: Sun Mar 25, 2018 12:34 am
- Location: United States
Texture conversion and caching question
I've been doing a little programming for the fine folks working on Redguard Unity and we had a question about Daggerfall Unity's texture loading process. From what I can tell, DFU converts textures vanilla textures and then caches them for a period of time in memory. Is there are a specific reason why we don't cache converted textures to disk as PNGs so that we can skip the conversion process? I'm assuming it's because there wouldn't be an appreciable difference in load times. Am I correct or is there another reason?
Nexus DFU Mods:
https://www.nexusmods.com/daggerfalluni ... user+files
Mod source repo:
https://github.com/numidium/dfu-mods
https://www.nexusmods.com/daggerfalluni ... user+files
Mod source repo:
https://github.com/numidium/dfu-mods
- Kab the Bird Ranger
- Posts: 123
- Joined: Tue Feb 23, 2021 12:30 am
Re: Texture conversion and caching question
It could be cached to disk, but then we'd have to know when to invalidate the cache. Mods can override those textures, so the system would have to know when to use the cache files and when to reload the "right texture".
Also, between loading a TES2 texture from disk and loading a PNG from disk... Both requiring accessing the disk, and both require conversions before being sent to the GPU. I'm not sure the performance would be that different.
Also, between loading a TES2 texture from disk and loading a PNG from disk... Both requiring accessing the disk, and both require conversions before being sent to the GPU. I'm not sure the performance would be that different.
- numidium3rd
- Posts: 188
- Joined: Sun Mar 25, 2018 12:34 am
- Location: United States
Re: Texture conversion and caching question
Thanks for the reply. I agree, there's probably very little performance difference between streaming in a native texture versus a PNG. I honestly should have thought this through better before posting lol.
Nexus DFU Mods:
https://www.nexusmods.com/daggerfalluni ... user+files
Mod source repo:
https://github.com/numidium/dfu-mods
https://www.nexusmods.com/daggerfalluni ... user+files
Mod source repo:
https://github.com/numidium/dfu-mods
- Interkarma
- Posts: 7253
- Joined: Sun Mar 22, 2015 1:51 am
Re: Texture conversion and caching question
Hey Numidium.
Our gamedata isn't identical, so my experience might be of little value here, but there were some good reasons for DFU to use runtime conversion over pre-converting and caching everything beforehand.
First up, it's pretty inexpensive either way. It's still fast to read in a small file and expand it memory and it saves the hassle of maintaining cache differences between builds. Your target formats will probably change a bit as you don't know what you don't know yet.
This key one is that Daggerfall stores more data with textures than just images. It also stores scaling and offsets for world flats and position data for paper doll textures. This doesn't translate well to PNG so the original files need to be read anyway (or this data or also pre-cached to XML or something). And some images (e.g. weapons) are held in an indexed format for palette swaps that also doesn't translate well to PNG.
Also keep in mind that DFU started as Daggerfall Tools for Unity, a gamedata importer based on my old exploring tools. Part of this process is just an evolution from one thing to another. Personally, I think there's a niceness to just running the game and everything loading straight from classic gamedata without user needing to build a cache or re-build when formats change. The hard part of getting people to use good-quality gamedata in the first place.
Hopefully that's of some help, considering our requirements are probably different between the two games. I reckon just do whatever you think is best and let it grow from there. Good luck!
Our gamedata isn't identical, so my experience might be of little value here, but there were some good reasons for DFU to use runtime conversion over pre-converting and caching everything beforehand.
First up, it's pretty inexpensive either way. It's still fast to read in a small file and expand it memory and it saves the hassle of maintaining cache differences between builds. Your target formats will probably change a bit as you don't know what you don't know yet.
This key one is that Daggerfall stores more data with textures than just images. It also stores scaling and offsets for world flats and position data for paper doll textures. This doesn't translate well to PNG so the original files need to be read anyway (or this data or also pre-cached to XML or something). And some images (e.g. weapons) are held in an indexed format for palette swaps that also doesn't translate well to PNG.
Also keep in mind that DFU started as Daggerfall Tools for Unity, a gamedata importer based on my old exploring tools. Part of this process is just an evolution from one thing to another. Personally, I think there's a niceness to just running the game and everything loading straight from classic gamedata without user needing to build a cache or re-build when formats change. The hard part of getting people to use good-quality gamedata in the first place.
Hopefully that's of some help, considering our requirements are probably different between the two games. I reckon just do whatever you think is best and let it grow from there. Good luck!
- numidium3rd
- Posts: 188
- Joined: Sun Mar 25, 2018 12:34 am
- Location: United States
Re: Texture conversion and caching question
Thanks a bunch for the detailed post, Interkarma. I didn't know that Daggerfall textures had all that extra data in them. Redguard's texture files are just bitmaps with simple headers as far as we can tell (except for animated textures - I'm still working on reversing those). I agree that it's best to have the fewest steps possible between reading a vanilla file and instantiating a Unity texture. Fewer things can go wrong in translation as you mentioned.
And thanks for the well-wishes. You're an inspiration to Elder Scrolls game hackers everywhere .
And thanks for the well-wishes. You're an inspiration to Elder Scrolls game hackers everywhere .
Nexus DFU Mods:
https://www.nexusmods.com/daggerfalluni ... user+files
Mod source repo:
https://github.com/numidium/dfu-mods
https://www.nexusmods.com/daggerfalluni ... user+files
Mod source repo:
https://github.com/numidium/dfu-mods