[MOD] Distant Terrain

[pango]

After some experimentation, adding memory pressure works, but is probably burning a lot of CPU, that's not efficient at all.
I suspect that DaggerfallTerrain.DisposeNativeMemory() must not always be called explicitly in some paths, and that we rely on DaggerfallTerrain destructor being called to dispose of native arrays...

[Nystul]

I will try the profiler on my windows system and see if I can find out something

[pango]

With plain Debug.Log()s, it seems my hypothesis is wrong, I've yet to see native memory released from ~DaggerfallTerrain().
However it seems the game can simply be overwhelmed with jobs, when I stop moving around I see batches of DisposeNativeMemory() calls. And when the system starts to swap memory, it's all over because jobs get even slower to process.
Something else I noticed: I've been traveling SW in Sentinel, I don't remember moving backward, and yet in my traces I see native arrays being disposed several times for the same mapPixel, up to 3 times and even a few 4 times

Code: Select all

      1 DaggerfallTerrain dispose native memory MapX=381 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=351
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=354
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=382 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=354
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=355
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=382 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=351
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=384 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=384 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=385 MapY=345
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=385 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=386 MapY=344
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=345
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=386 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=387 MapY=343
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=344
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=345
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=350
      4 DaggerfallTerrain dispose native memory MapX=387 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=387 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=388 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=388 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=388 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=389 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=350
      4 DaggerfallTerrain dispose native memory MapX=389 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=352
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=353
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=354
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=355
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=389 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=390 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=352
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=353
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=355
      1 DaggerfallTerrain dispose native memory MapX=390 MapY=356
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=346
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=347
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=348
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=349
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=354
      1 DaggerfallTerrain dispose native memory MapX=392 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=392 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=392 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=392 MapY=354
      1 DaggerfallTerrain dispose native memory MapX=393 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=393 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=393 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=394 MapY=343
      3 DaggerfallTerrain dispose native memory MapX=394 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=394 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=395 MapY=344
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=345
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=346
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=347
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=348
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=349
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=395 MapY=351
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=352

Is it because of transition ring, or could it be the sign of some issue?

[pango]

Okay, some good news. But I should really have thought about doing tests with just that mod installed
Without RealGrass mod that was slowing down my system, I can tedious travel around at 10x with the alpha version of Distant Terrain just fine. So it looks like the mod is not buggy on its own, however the way terrain generation works, if your system is not fast enough to dequeue the jobs faster than they arrive, bad things start to happen...

[Nystul]

Okay, some good news. But I should really have thought about doing tests with just that mod installed
Without RealGrass mod that was slowing down my system, I can tedious travel around at 10x with the alpha version of Distant Terrain just fine. So it looks like the mod is not buggy on its own, however the way terrain generation works, if your system is not fast enough to dequeue the jobs faster than they arrive, bad things start to happen...

ok that are partly good news.

Nevertheless I experience slow but steady growing memory with distant terrain being the only active mod, which is no good sign.

I tested yesterday dfunity without any mods seems to behave good now (although it seems to me it is still leaking a bit (neglectable) since memory will go from initial about 1gb on my system to 1.3gb if I often fast travel and walk around alot with speed cheat - it is hard to tell since it will also go down depending on how many locations are loaded at the same time - but there seems to be a general upward trend)

Distant Terrain on the other hand causes memory to go up faster and I see why it would eventuelly crash due to out of memory problems.

Today I have some time to test more. So maybe I can catch something, furthermore I want to test a version with DaggerfallTerrain.DisposeNativeMemory() on map pixel change

[pango]

[...] furthermore I want to test a version with DaggerfallTerrain.DisposeNativeMemory() on map pixel change

I'm not entirely sure what you have in mind, but it seems you can only do something with the native arrays once the job that references them is completed; And it seems there's no way to cancel a scheduled job either.
So unless you somehow pospone some jobs scheduling, you can't dispose of more native memory than you can from the code after jobHandle.Complete().
I also thought about starting jobs by checking of the map pixel being worked on is still within player's range. Problem, you're not supposed to access data outside of jobs' context in general, so it may not be possible.

That job system doesn't seem to lend itself easily to "potentially useful computations"...

Given that terrain sampler is already multithreaded with IJobParallelFor, maybe we should only schedule one (or at most a few) map pixel at a time?

[Nystul]

ah sry, i meant DaggerfallGC.ThrottledUnloadUnusedAssets()

one interesting finding is, when running dfunity from editor with mod active, all that memory that grows while playing does not get released even when I stop execution - so the unity process consumes more and more memory - I would have expected the memory to get released when I stop the execution in unity editor but no...

Does this indicate that the memory leak is leaking of native memory or can GC just not free managed memory for some reason here?

update: ok, so also dfunity without mods ends up with unity editor process memory not being released - so this seems to be "normal behaviour"

[pango]

Ok, I found a reproducible way to trigger "missing town" bug.
If I do a round-trip between Linisa and Isfakabi (Sentinel), then the initial town will have disappeared.

Terrain Distance = 4
Linisa mappixel = [391,347]
Isfakabi mappixel = [392,351]

It does not matter what town I start from.. It happens with both stable and alpha versions of the mod (and not with the default terrain generator).
I tested it using Tedious Travel, but it can be reproduced without this mod (it just takes more time).

linisa.jpg (220.74 KiB) Viewed 2168 times

[Nystul]

thanks for reporting - will give it a try!
does it happen only with distant terrain mod enabled? or is it just making it more likely to occur?

update: yeah, I can reproduce it

update2: first finding ist this: There remain terrain gameobjects that are named "Pooled" in "StreamingTarget" gameobject behind. They are terrains that are no longer used but are still in hierarchy. A further indicator that those are abandoned are that their height differs from current terrain height - I guess this is due to floating origin updates to maintained terrain that of course don't update those abandoned terrains. Another finding is that those terrains are still neighbors to active terrains
Will further investigate..



update3: ok so "Pooled" terrains often don't have any active neighbor terrains (if so, "Pooled" gameobject is grayed out) - sometimes though "Pooled" terrains contain some active terrains as neighbors (in this case those are not grayed out)
Could it be possible that those terrains are reused when round-trippin but don't have any buildings left (since they have already been destroyed) and we end up without buildings in that location?

travelling further in the same direction seems to make "Pooled" terrains with neighbors into grayed-out "Pooled" terrains eventually (which get disposed eventually) - but they exist for I guess on map pixel traversal. So if TerrainDistance is just right they don't get released on round-trips. I guess this reuse was intended but maybe we forgot to reload locations in such case?

It happens with both stable and alpha versions of the mod (and not with the default terrain generator).

Distant Terrain is obfuscating stuff a bit since it changes TerrainDistance setting by 1 - so maybe on the "matching" setting it would also occur with default terrain sampler. My feeling is that it does not have much to do with terrain sampler at all but rather with terrain generation/maintaining

update4: yes, confirmed. Set terrain distance to 3 and do the round-trip (Linisa to Isfakabi and back) in dfunity without mods and you will get the same reproducible behaviour

[pango]

update4: yes, confirmed. Set terrain distance to 3 and do the round-trip (Linisa to Isfakabi and back) in dfunity without mods and you will get the same reproducible behaviour

That's interesting, maybe we're on the right track to fix the more general bug!
(Yup, just saw you referenced this thread)