[MOD] Distant Terrain

A curated forum for compatible and maintained mods. Users are unable to create new topics in this forum but can reply to existing topics. Please message a moderator to have your mod moved into this forum area.
Post Reply
User avatar
pango
Posts: 2473
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: [MOD] Distant Terrain

Post by pango » Sun May 31, 2020 2:45 pm

After some experimentation, adding memory pressure works, but is probably burning a lot of CPU, that's not efficient at all.
I suspect that DaggerfallTerrain.DisposeNativeMemory() must not always be called explicitly in some paths, and that we rely on DaggerfallTerrain destructor being called to dispose of native arrays...
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
Nystul
Posts: 1432
Joined: Mon Mar 23, 2015 8:31 am

Re: [MOD] Distant Terrain

Post by Nystul » Sun May 31, 2020 2:50 pm

I will try the profiler on my windows system and see if I can find out something

User avatar
pango
Posts: 2473
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: [MOD] Distant Terrain

Post by pango » Sun May 31, 2020 7:37 pm

With plain Debug.Log()s, it seems my hypothesis is wrong, I've yet to see native memory released from ~DaggerfallTerrain().
However it seems the game can simply be overwhelmed with jobs, when I stop moving around I see batches of DisposeNativeMemory() calls. And when the system starts to swap memory, it's all over because jobs get even slower to process.
Something else I noticed: I've been traveling SW in Sentinel, I don't remember moving backward, and yet in my traces I see native arrays being disposed several times for the same mapPixel, up to 3 times and even a few 4 times

Code: Select all

      1 DaggerfallTerrain dispose native memory MapX=381 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=351
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=354
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=381 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=381 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=382 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=354
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=355
      3 DaggerfallTerrain dispose native memory MapX=382 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=382 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=382 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=351
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=383 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=383 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=384 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=384 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=384 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=384 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=385 MapY=345
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=385 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=385 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=385 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=386 MapY=344
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=345
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=386 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=386 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=386 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=387 MapY=343
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=344
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=345
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=346
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=350
      4 DaggerfallTerrain dispose native memory MapX=387 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=387 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=387 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=387 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=388 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=388 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=353
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=355
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=388 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=388 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=389 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=348
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=349
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=350
      4 DaggerfallTerrain dispose native memory MapX=389 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=352
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=353
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=354
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=355
      3 DaggerfallTerrain dispose native memory MapX=389 MapY=356
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=357
      2 DaggerfallTerrain dispose native memory MapX=389 MapY=358
      1 DaggerfallTerrain dispose native memory MapX=389 MapY=359
      1 DaggerfallTerrain dispose native memory MapX=390 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=347
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=352
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=353
      3 DaggerfallTerrain dispose native memory MapX=390 MapY=354
      2 DaggerfallTerrain dispose native memory MapX=390 MapY=355
      1 DaggerfallTerrain dispose native memory MapX=390 MapY=356
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=346
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=347
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=348
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=349
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=391 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=391 MapY=354
      1 DaggerfallTerrain dispose native memory MapX=392 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=392 MapY=351
      3 DaggerfallTerrain dispose native memory MapX=392 MapY=352
      2 DaggerfallTerrain dispose native memory MapX=392 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=392 MapY=354
      1 DaggerfallTerrain dispose native memory MapX=393 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=393 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=393 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=393 MapY=353
      1 DaggerfallTerrain dispose native memory MapX=394 MapY=343
      3 DaggerfallTerrain dispose native memory MapX=394 MapY=344
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=345
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=346
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=347
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=348
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=349
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=350
      3 DaggerfallTerrain dispose native memory MapX=394 MapY=351
      2 DaggerfallTerrain dispose native memory MapX=394 MapY=352
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=343
      2 DaggerfallTerrain dispose native memory MapX=395 MapY=344
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=345
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=346
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=347
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=348
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=349
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=350
      2 DaggerfallTerrain dispose native memory MapX=395 MapY=351
      1 DaggerfallTerrain dispose native memory MapX=395 MapY=352
Is it because of transition ring, or could it be the sign of some issue?
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
pango
Posts: 2473
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: [MOD] Distant Terrain

Post by pango » Sun May 31, 2020 11:52 pm

Okay, some good news. But I should really have thought about doing tests with just that mod installed :oops:
Without RealGrass mod that was slowing down my system, I can tedious travel around at 10x with the alpha version of Distant Terrain just fine. So it looks like the mod is not buggy on its own, however the way terrain generation works, if your system is not fast enough to dequeue the jobs faster than they arrive, bad things start to happen...
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
Nystul
Posts: 1432
Joined: Mon Mar 23, 2015 8:31 am

Re: [MOD] Distant Terrain

Post by Nystul » Mon Jun 01, 2020 9:47 am

pango wrote:
Sun May 31, 2020 11:52 pm
Okay, some good news. But I should really have thought about doing tests with just that mod installed :oops:
Without RealGrass mod that was slowing down my system, I can tedious travel around at 10x with the alpha version of Distant Terrain just fine. So it looks like the mod is not buggy on its own, however the way terrain generation works, if your system is not fast enough to dequeue the jobs faster than they arrive, bad things start to happen...
ok that are partly good news.

Nevertheless I experience slow but steady growing memory with distant terrain being the only active mod, which is no good sign.

I tested yesterday dfunity without any mods seems to behave good now (although it seems to me it is still leaking a bit (neglectable) since memory will go from initial about 1gb on my system to 1.3gb if I often fast travel and walk around alot with speed cheat - it is hard to tell since it will also go down depending on how many locations are loaded at the same time - but there seems to be a general upward trend)

Distant Terrain on the other hand causes memory to go up faster and I see why it would eventuelly crash due to out of memory problems.

Today I have some time to test more. So maybe I can catch something, furthermore I want to test a version with DaggerfallTerrain.DisposeNativeMemory() on map pixel change

User avatar
pango
Posts: 2473
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: [MOD] Distant Terrain

Post by pango » Mon Jun 01, 2020 11:32 am

Nystul wrote:
Mon Jun 01, 2020 9:47 am
[...] furthermore I want to test a version with DaggerfallTerrain.DisposeNativeMemory() on map pixel change
I'm not entirely sure what you have in mind, but it seems you can only do something with the native arrays once the job that references them is completed; And it seems there's no way to cancel a scheduled job either.
So unless you somehow pospone some jobs scheduling, you can't dispose of more native memory than you can from the code after jobHandle.Complete().
I also thought about starting jobs by checking of the map pixel being worked on is still within player's range. Problem, you're not supposed to access data outside of jobs' context in general, so it may not be possible.

That job system doesn't seem to lend itself easily to "potentially useful computations"...

Given that terrain sampler is already multithreaded with IJobParallelFor, maybe we should only schedule one (or at most a few) map pixel at a time?
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
Nystul
Posts: 1432
Joined: Mon Mar 23, 2015 8:31 am

Re: [MOD] Distant Terrain

Post by Nystul » Mon Jun 01, 2020 2:00 pm

ah sry, i meant DaggerfallGC.ThrottledUnloadUnusedAssets()

one interesting finding is, when running dfunity from editor with mod active, all that memory that grows while playing does not get released even when I stop execution - so the unity process consumes more and more memory - I would have expected the memory to get released when I stop the execution in unity editor but no...

Does this indicate that the memory leak is leaking of native memory or can GC just not free managed memory for some reason here?

update: ok, so also dfunity without mods ends up with unity editor process memory not being released - so this seems to be "normal behaviour"

User avatar
pango
Posts: 2473
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: [MOD] Distant Terrain

Post by pango » Tue Jun 02, 2020 9:13 pm

Ok, I found a reproducible way to trigger "missing town" bug.
If I do a round-trip between Linisa and Isfakabi (Sentinel), then the initial town will have disappeared.

Terrain Distance = 4
Linisa mappixel = [391,347]
Isfakabi mappixel = [392,351]

It does not matter what town I start from.. It happens with both stable and alpha versions of the mod (and not with the default terrain generator).
I tested it using Tedious Travel, but it can be reproduced without this mod (it just takes more time).
linisa.jpg
linisa.jpg (220.74 KiB) Viewed 347 times
Attachments
SAVE548.zip
(464.77 KiB) Downloaded 6 times
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
Nystul
Posts: 1432
Joined: Mon Mar 23, 2015 8:31 am

Re: [MOD] Distant Terrain

Post by Nystul » Wed Jun 03, 2020 7:26 am

thanks for reporting - will give it a try!
does it happen only with distant terrain mod enabled? or is it just making it more likely to occur?

update: yeah, I can reproduce it

update2: first finding ist this: There remain terrain gameobjects that are named "Pooled" in "StreamingTarget" gameobject behind. They are terrains that are no longer used but are still in hierarchy. A further indicator that those are abandoned are that their height differs from current terrain height - I guess this is due to floating origin updates to maintained terrain that of course don't update those abandoned terrains. Another finding is that those terrains are still neighbors to active terrains
Will further investigate..

Image

update3: ok so "Pooled" terrains often don't have any active neighbor terrains (if so, "Pooled" gameobject is grayed out) - sometimes though "Pooled" terrains contain some active terrains as neighbors (in this case those are not grayed out)
Could it be possible that those terrains are reused when round-trippin but don't have any buildings left (since they have already been destroyed) and we end up without buildings in that location?

travelling further in the same direction seems to make "Pooled" terrains with neighbors into grayed-out "Pooled" terrains eventually (which get disposed eventually) - but they exist for I guess on map pixel traversal. So if TerrainDistance is just right they don't get released on round-trips. I guess this reuse was intended but maybe we forgot to reload locations in such case?
pango wrote:
Tue Jun 02, 2020 9:13 pm
It happens with both stable and alpha versions of the mod (and not with the default terrain generator).
Distant Terrain is obfuscating stuff a bit since it changes TerrainDistance setting by 1 - so maybe on the "matching" setting it would also occur with default terrain sampler. My feeling is that it does not have much to do with terrain sampler at all but rather with terrain generation/maintaining

update4: yes, confirmed. Set terrain distance to 3 and do the round-trip (Linisa to Isfakabi and back) in dfunity without mods and you will get the same reproducible behaviour

User avatar
pango
Posts: 2473
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: [MOD] Distant Terrain

Post by pango » Wed Jun 03, 2020 10:51 am

Nystul wrote:
Wed Jun 03, 2020 7:26 am
update4: yes, confirmed. Set terrain distance to 3 and do the round-trip (Linisa to Isfakabi and back) in dfunity without mods and you will get the same reproducible behaviour
That's interesting, maybe we're on the right track to fix the more general bug!
(Yup, just saw you referenced this thread)
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

Post Reply