[MOD] Distant Terrain

[Nystul]

can you tell us the map location (nearest town e.g.) from where that screenshot was captured? thank you

[herman2000]

Hm. With Distant Terrain's improved Terrain setting, I'm getting some weird results in south-Dak'fron. Fast-Travelling to Lothten Orchard spawns me beneath the Terrain. Going from The Queen's Huntsman's Lodge and running South East for a while, I'll eventually see giant holes in the terrain. With the setting turned off, all is well. Quite bizarre.

Spoiler!

[pango]

They're known issues going near the southern border indeed
viewtopic.php?f=27&t=10&start=440#p32197
viewtopic.php?f=27&t=10&start=410#p23689
viewtopic.php?f=4&t=37&start=350#p40893

[Nystul]

I took a look and it's not due to the jobs update. I did track down the error to these lines 606-608 of ImprovedTerrainSampler.cs:

Code: Select all

int mapPixelX = Math.Max(0, Math.Min(mx + x - 2, WoodsFile.mapWidthValue -1));
int mapPixelY = Math.Max(0, Math.Min(my + y - 2, WoodsFile.mapHeightValue -1));
waterDistanceMap[x, y] = (float)Math.Sqrt(ImprovedWorldTerrain.MapDistanceSquaredFromWater[mapPixelY * WoodsFile.mapWidthValue + mapPixelX]);

The array out of bounds happens when the mapPixelY becomes 500, so when multiplied by 1000 it's 500,000 which is the length of the array. Subtracting 1 from the map width and height values (as shown in the modified code above) stops the OOB but it may be that the actual mapPixelY/X need 1 subtracting for the array. I will leave that to you Nystul as I have no idea what this code is actually doing.

thanks for investigating - this is indeed an easy fix - thanks for your efforts!

update: there is a new version 2.6.1 with retro rendering mode support and map border fix in first post of this thread

[Nystul]

Another smaller issue, probably already known, is the small trenches at the zone borders, mostly noticeable in rough terrains...
trenches.jpg
trenches 2.jpg

I found time to improve the heights generation in the terrain sampler so that neighboring map pixels with different climate (and thus different height noise parameters) are blended into each other making the "trench" workaround obsolete.

I will also try to get rid of the Resources.UnloadUnusedAssets() calls - performance is better without (less hickups) - we have to test and wait for feedback if some people experience crashes again but if we need it I plan to implement a mod setting to disable/enable it - but lets try to get along without it

I might have found the issue with the crash on exit as well

link to alpha build "Distant Terrain 2.7.0a" updated on first post of this thread - pls test and report back how it works for you

update: here is a comparison screenshot regarding the trenches (mysterious tree spawned):

[BadLuckBurt]

Looking good, I'm going to sift through the code to see how you did the trench fix. I'm close to finishing my own terrain sampler which has climate merging as well but the code could probably more elegant.

I'm also curious, do you use different types of noise for climates or is it just the scaling settings that are changed?

I ended up on this blog at some point: https://www.redblobgames.com/maps/terra ... #elevation while doing research and incorporated some of those functions into mine to give it more variation.

[Nystul]

I preprocess the small height map from daggerfall to make it more realistic (e.g. heights of map pixels depend on distance from water and climate)

baseheight from updated small height map (bicubic interpolation between map pixels) multiplied by a factor (=main height) + noise height from large heightmap (bicubic interpolation again) multiplied by a factor (extra height variation) + small extra noise height (small terrain spikes) multiplied by a factor depended on climate (bilinear interpolation between map pixels).

bilinear interpolation between map pixels for climate does work like this:
there is a height multiplicator for every climate
for every map pixel I compute:

• a "top left height multiplier" as average between neighbor to the top left, to the top, to the left and current map pixel
• a "top right height multiplier" as average between neighbor to the top, current map pixel, neighbor to the top right, and right
• a "bottom left height multiplier" as average between neighbor to the left, current map pixel, neighbor to the bottom left, and bottom
• a "bottom right height multiplier" as average between current map pixel, neighbor to the right, and bottom, and bottom right

bilinear interpolation is done between these 2x2 values

I'm also curious, do you use different types of noise for climates or is it just the scaling settings that are changed?

yes, different types of noise for climates - that caused the problems in the first place when 2 neighboring map pixels had different climate

I ended up on this blog at some point: https://www.redblobgames.com/maps/terra ... #elevation while doing research and incorporated some of those functions into mine to give it more variation.

cool, really interested in what your terrain sampler will look like

[pango]

Glad to see this mod be worked on again!
So, I tested the alpha,
The good:

• works fine under Linux
• with retro mode support too, yay!
• I checked the world borders, North, Each and South, everything seems fine
• I like the improvements on terrain generation itself

The bad:

• I had a few cases of "missing towns" with traveling around on horse (without Tedious Travel, just plain moving around), but that's a problem that is known to happen already so it's hard to tell if that mod version really has an impact on that problem.
• I hate to be this guy, but the last problem to report is high memory usage when the mod is stressed, like when using Tedious Travel. As probably expected, it gets worse with high Terrain Distance; While tedious traveling with a Terrain Distance = 4, I've seen memory usage of almost 21GB with no texture mod (then the game just exited, no log message).

I don't know if Resources.UnloadUnusedAssets() (or the GC.Collect() call it contains) are absolutely necessary, or is hiding some issue in the way memory is used (jobs, native arrays,...).

Also as I side note, you can try using DaggerfallGC.ThrottledUnloadUnusedAssets() instead of directly calling Resources.UnloadUnusedAssets() so that collection does not happen more than once every 3 minutes. That may be a bit too slow when used with high time compression tedious travel though, but one benefit vs. reimplementing the same logic in each mod is that the timer is shared between all throttled calls, so collections happen the least number of times. Maybe the API should be modified so one can specify the minimum delay between collections? eg. ThrottledUnloadUnusedAssets(int minDelay)
But I really hope forced garbage collection can be avoided altogether... (native arrays pooling?)

[Nystul]

I don't know if Resources.UnloadUnusedAssets() (or the GC.Collect() call it contains) are absolutely necessary, or is hiding some issue in the way memory is used (jobs, native arrays,...).

version 2.7.0a does not have any of these - so I now know it was in there for a reason :/

I really need to find out what leaks all this memory. earlier tests showed that even without any mods dfunity leaks memory on fast travel/map pixel changes. But it is less and thus much later problems start to occur. And this might have been already fixed - have to give it another try
Need to learn that profiler feature of unity I guess and take a look.
I always thought GC should clean up all data which is not referenced anywhere anymore - but I think something is not working as intended at least in distant terrain mod. I think some of the terrains are not correctly deleted for some reason.

[pango]

I always thought GC should clean up all data which is not referenced anywhere anymore - but I think something is not working as intended at least in distant terrain mod.

Well, that's what garbage collectors do eventually for all managed memory.

Unreleased managed memory can come from some dangling reference, the object not released could be in some container somewhere, that is itself considered a "live" object so it's not released. Since calling GC.Collect() (thru Resources.UnloadUnusedAssets()) seems to fix the issue, I guess it's not the case.

Unmanaged memory: Distant Terrain is handling native memory thru NativeArrays, and I suspect jobs themselves need more native memory internally. But they should be all fenced by managed objects that should released the native memory when their Dispose() method is called (which should be automatic) and all should be fine. Again, the fact that calling GC.Collect() keeps memory under control means there should be no bug with native memory handling.

Eventual collection: garbage collection happens concurrently or in parallel to the main program, and how fast it scans memory for objects to reclaim is adjusted using an estimate of how fast garbage is being produced. Since most of the memory used is native memory, the garbage collector can be made aware of allocations outside of managed memory using AddMemoryPressure() and RemoveMemoryPressure(). I wonder if NativeArray does that already, or if it needs to be done explicitly.

Some of this is best guesses, maybe we're hitting a bug in some external code for example...
Indeed the next step is using the profiler (Window > Analysis > Profiler). Doesn't work too well under Linux though, I can't resize its window to begin with

profiler.jpg (15.4 KiB) Viewed 1629 times