[Updated] Caching map pixel height map samples

[BadLuckBurt]

Hi everyone,

I'm going to run an experiment by caching the generated height map samples in my mod's persistent data folder and loading those instead of recalculating the values each time a map pixel gets loaded.

I know the disk space requirements are going to be ridiculous (25GB) for all 500.000 map pixels but I'm hoping the speed gain will be worth it. Seeing how modern games are freaking huge when it comes to filesize, I imagine there are people who would turn this cache on since all it would do is increase your install size slightly

- Update 1 -

So, after 20 something hours I've finally generated the 500.000 files for the whole map. Total file size is 24.1GB uncompressed.

I have tested the Travel Options-mod in the Wrothgarian Mountains and I am able to move with the speed / timescale cranked up to 40x. I have not fallen through the terrain and the game has not crashed.

Can anyone run this mod with 40x speed normally? I know my rig just goes "I don't think so, Tim" when I try that without this cache.

The average terrains update time is about 500ms. I've extract the times from my test run:

Code: Select all

	Line 2022: DFTFU 1.9.1: Time to update terrains: 3684ms 
	Line 2154: DFTFU 1.9.1: Time to update terrains: 398ms 
	Line 2205: DFTFU 1.9.1: Time to update terrains: 498ms 
	Line 2256: DFTFU 1.9.1: Time to update terrains: 443ms 
	Line 2343: DFTFU 1.9.1: Time to update terrains: 455ms 
	Line 2355: DFTFU 1.9.1: Time to update terrains: 517ms 
	Line 2376: DFTFU 1.9.1: Time to update terrains: 477ms 
	Line 2388: DFTFU 1.9.1: Time to update terrains: 517ms 
	Line 2400: DFTFU 1.9.1: Time to update terrains: 525ms 
	Line 2415: DFTFU 1.9.1: Time to update terrains: 503ms 
	Line 2439: DFTFU 1.9.1: Time to update terrains: 500ms 
	Line 2454: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 2469: DFTFU 1.9.1: Time to update terrains: 519ms 
	Line 2481: DFTFU 1.9.1: Time to update terrains: 507ms 
	Line 2496: DFTFU 1.9.1: Time to update terrains: 499ms 
	Line 2508: DFTFU 1.9.1: Time to update terrains: 511ms 
	Line 2529: DFTFU 1.9.1: Time to update terrains: 510ms 
	Line 2544: DFTFU 1.9.1: Time to update terrains: 494ms 
	Line 2556: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 2571: DFTFU 1.9.1: Time to update terrains: 461ms 
	Line 2583: DFTFU 1.9.1: Time to update terrains: 499ms 
	Line 2595: DFTFU 1.9.1: Time to update terrains: 487ms 
	Line 2607: DFTFU 1.9.1: Time to update terrains: 498ms 
	Line 2622: DFTFU 1.9.1: Time to update terrains: 505ms 
	Line 2637: DFTFU 1.9.1: Time to update terrains: 519ms 
	Line 2661: DFTFU 1.9.1: Time to update terrains: 527ms 
	Line 2676: DFTFU 1.9.1: Time to update terrains: 528ms 
	Line 2694: DFTFU 1.9.1: Time to update terrains: 459ms 
	Line 2706: DFTFU 1.9.1: Time to update terrains: 492ms 
	Line 2721: DFTFU 1.9.1: Time to update terrains: 541ms 
	Line 2739: DFTFU 1.9.1: Time to update terrains: 443ms 
	Line 2751: DFTFU 1.9.1: Time to update terrains: 488ms 
	Line 2763: DFTFU 1.9.1: Time to update terrains: 529ms 
	Line 2775: DFTFU 1.9.1: Time to update terrains: 509ms 
	Line 2787: DFTFU 1.9.1: Time to update terrains: 527ms 
	Line 2802: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 2814: DFTFU 1.9.1: Time to update terrains: 508ms 
	Line 2829: DFTFU 1.9.1: Time to update terrains: 527ms 
	Line 2844: DFTFU 1.9.1: Time to update terrains: 538ms 
	Line 2862: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 2874: DFTFU 1.9.1: Time to update terrains: 492ms 
	Line 2886: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 2901: DFTFU 1.9.1: Time to update terrains: 545ms 
	Line 2916: DFTFU 1.9.1: Time to update terrains: 503ms 
	Line 2928: DFTFU 1.9.1: Time to update terrains: 487ms 
	Line 2940: DFTFU 1.9.1: Time to update terrains: 547ms 
	Line 2955: DFTFU 1.9.1: Time to update terrains: 510ms 
	Line 2967: DFTFU 1.9.1: Time to update terrains: 515ms 
	Line 2979: DFTFU 1.9.1: Time to update terrains: 540ms 
	Line 2994: DFTFU 1.9.1: Time to update terrains: 522ms 
	Line 3006: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 3021: DFTFU 1.9.1: Time to update terrains: 560ms 
	Line 3036: DFTFU 1.9.1: Time to update terrains: 535ms 
	Line 3048: DFTFU 1.9.1: Time to update terrains: 516ms 
	Line 3060: DFTFU 1.9.1: Time to update terrains: 509ms 
	Line 3072: DFTFU 1.9.1: Time to update terrains: 537ms 
	Line 3084: DFTFU 1.9.1: Time to update terrains: 526ms 
	Line 3099: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 3111: DFTFU 1.9.1: Time to update terrains: 515ms 
	Line 3123: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 3152: DFTFU 1.9.1: Time to update terrains: 756ms 
	Line 3164: DFTFU 1.9.1: Time to update terrains: 516ms 
	Line 3176: DFTFU 1.9.1: Time to update terrains: 511ms 
	Line 3188: DFTFU 1.9.1: Time to update terrains: 520ms 
	Line 3200: DFTFU 1.9.1: Time to update terrains: 521ms 
	Line 3212: DFTFU 1.9.1: Time to update terrains: 526ms 
	Line 3227: DFTFU 1.9.1: Time to update terrains: 592ms 
	Line 3239: DFTFU 1.9.1: Time to update terrains: 579ms 
	Line 3251: DFTFU 1.9.1: Time to update terrains: 499ms 
	Line 3263: DFTFU 1.9.1: Time to update terrains: 507ms 
	Line 3293: DFTFU 1.9.1: Time to update terrains: 522ms 
	Line 3353: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 3365: DFTFU 1.9.1: Time to update terrains: 517ms 
	Line 3377: DFTFU 1.9.1: Time to update terrains: 510ms 
	Line 3389: DFTFU 1.9.1: Time to update terrains: 510ms 
	Line 3404: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 3416: DFTFU 1.9.1: Time to update terrains: 502ms 
	Line 3428: DFTFU 1.9.1: Time to update terrains: 512ms 
	Line 3440: DFTFU 1.9.1: Time to update terrains: 513ms 
	Line 3452: DFTFU 1.9.1: Time to update terrains: 506ms 
	Line 3476: DFTFU 1.9.1: Time to update terrains: 517ms 
	Line 3488: DFTFU 1.9.1: Time to update terrains: 541ms 
	Line 3503: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 3515: DFTFU 1.9.1: Time to update terrains: 499ms 
	Line 3527: DFTFU 1.9.1: Time to update terrains: 514ms 
	Line 3539: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 3551: DFTFU 1.9.1: Time to update terrains: 547ms 
	Line 3566: DFTFU 1.9.1: Time to update terrains: 527ms 
	Line 3593: DFTFU 1.9.1: Time to update terrains: 501ms 
	Line 3605: DFTFU 1.9.1: Time to update terrains: 522ms 
	Line 3617: DFTFU 1.9.1: Time to update terrains: 500ms 
	Line 3632: DFTFU 1.9.1: Time to update terrains: 0ms 
	Line 3647: DFTFU 1.9.1: Time to update terrains: 0ms 
	Line 3662: DFTFU 1.9.1: Time to update terrains: 0ms 
	Line 3677: DFTFU 1.9.1: Time to update terrains: 0ms 
	Line 3692: DFTFU 1.9.1: Time to update terrains: 0ms 
	Line 3707: DFTFU 1.9.1: Time to update terrains: 0ms 
	Line 3722: DFTFU 1.9.1: Time to update terrains: 0ms 
	Line 3737: DFTFU 1.9.1: Time to update terrains: 507ms 
	Line 3749: DFTFU 1.9.1: Time to update terrains: 467ms 
	Line 3764: DFTFU 1.9.1: Time to update terrains: 494ms 
	Line 3814: DFTFU 1.9.1: Time to update terrains: 510ms 
	Line 3829: DFTFU 1.9.1: Time to update terrains: 532ms 
	Line 3841: DFTFU 1.9.1: Time to update terrains: 509ms 
	Line 3853: DFTFU 1.9.1: Time to update terrains: 535ms 
	Line 3865: DFTFU 1.9.1: Time to update terrains: 540ms 
	Line 3880: DFTFU 1.9.1: Time to update terrains: 542ms 
	Line 3892: DFTFU 1.9.1: Time to update terrains: 506ms 

The first one (3684ms) is where I just booted up the game and loaded a save game. The rest are from the fast travel with Travel Options.

I'm testing this with a TerrainDistance of 6. I know the regular DFU download only goes up to TerrainDistance 4 but I've found 6 to be quite stable on my machine when moving at normal speed through the world without the cache.

[Macadaynu]

Cool let us know how it goes,

I would 100% trade a bigger install size for a performance increase

My main drops in FPS are when travelling with Travel Options using accelerated travel, just after completing the accelerated travel. Hopefully the caching could alleviate some of this slowdown.

[BadLuckBurt]

Cool let us know how it goes,

I would 100% trade a bigger install size for a performance increase

My main drops in FPS are when travelling with Travel Options using accelerated travel, just after completing the accelerated travel. Hopefully the caching could alleviate some of this slowdown.

That's one of the reasons I'm giving this a shot. It should help speed up the generation process. The way I'm doing it now is kinda hacky because I don't want to break the scheduled job system Hazelnut implemented.

At the moment, all the jobs that calculate the samples still run but instead of doing the actual calculations, I just set the height value for each sample and skip the rest of the code. Ideally, after I have this up and running, I'd like to just assign the array in one go and return the job immediately without running it but I haven't figured out how to do that yet.

What I have running so far:

In DaggerfallTerrain.cs, the array values are exported from CompleteMapPixelDataUpdate to a file in my mod's persistent data folder if a file does not exist yet for the map pixel coordinates it's processing.

I was forced to do this in a rather hacky way because I didn't manage to get an instance from my mod class so I'm using a hard-coded path for now. I think this has to do with the fact that DFU is compiled separately from the mod. I'm a big noob when it comes to compiled code coming from web development where things 'just work'. I tried several approaches but did not manage to get an instance, all because of my noobness probably but never stopped me.

I have an idea how to work around it but that would involve adding a new event to DaggerfallTerrain.cs, something like CacheMapPixelEvent that a mod can subscribe to, then I can just use that to access the array by subscribing to the event with my instantiated class and let that handle the rest.

Loading the exported array back into DFU is already handled by my mod's terrain sampler so if I can do the above through an event, I won't have to resort to this hacky stuff.

I should add that my mod uses less samples per terrain than the default sampler and probably the other terrain sampler's out there. The default one does 129 * 129 samples for one terrain so you'd get 16641 samples to store per file. Don't ask me why it does 129 instead of 128 but there's a reason for that related to the job system It just always adds 1 to your actual heightmap dimension.

My terrain sampler does 65 * 65 samples per terrain so I'm only store 4225 samples per file which comes to about 50KB per file. I decided to lower the amount of samples on mine to get a rougher looking terrain.

Since each terrain is 800x800 metres, a resolution of 128 means that one sample stretches 6.25 metres, I know modern games who would be jealous of that resolution considering the world size of Daggerfall. One sample on my terrain is 12.5 metres which is less subtle but I like it in this game.

Anyway, I wrote a script that teleports me through the entire world with a 5 second delay, I'm going to let that run overnight. I had it running for a few hours earlier but it only got to 13.000 map pixels. The way the code itself has been setup, I could just run around the world and play the game while it creates the files in the background but I want to see what happens when it runs with the full data set exported.

[Macadaynu]

Interesting, best of luck with this. Could hopefully help prevent players from falling through the world as the data are already cached and don't have to be regenerated.

[pango]

The default one does 129 * 129 samples for one terrain so you'd get 16641 samples to store per file. Don't ask me why it does 129 instead of 128 but there's a reason for that related to the job system

That looks more like a "poles and fences" problem, if a terrain is 128 units long, then it has 129 x 129 edges.
And yes, that means that adjacent terrains share edges at their common frontier.

[BadLuckBurt]

That looks more like a "poles and fences" problem, if a terrain is 128 units long, then it has 129 x 129 edges.
And yes, that means that adjacent terrains share edges at their common frontier.

You're right, the poles and fences bit made it click for me, the samplers calculate the poles, not the fences thank you for clearing that up

[Hazelnut]

I'm not sure this will make much difference to the terrain performance really, as it's multi-threaded and only one part of the chain. The parts that can't be parallelised are when creating unity game objects as that has to be done on the main thread, creating nature billboards and locations etc. It may have utility for anyone with plenty of ram but few processor cores though...

The FPS drops when using TO is often when travelling near towns etc, as their generation is done 1 block per frame and on the main thread. The issue with TO I want to solve is the falling off terrain and my testing last year got me to the point that the terrain event is sometimes missed when main thread is overloaded and so the terrain is not created and you fall off. I suspect this is only one cause, and there is another which I have not tracked down.

Oh and the 129 is nothing to do with jobs system, I re-implemented the algorithms without understanding them fully (at first) so wasn't changing anything except how the generation is executed.

[BadLuckBurt]

I disabled the cache and did roughly the same journey, it appears you're right, these times are only slightly higher on average:

Code: Select all

	Line 2134: DFTFU 1.9.1: Time to update terrains: 4167ms 
	Line 2302: DFTFU 1.9.1: Time to update terrains: 426ms 
	Line 2353: DFTFU 1.9.1: Time to update terrains: 529ms 
	Line 2404: DFTFU 1.9.1: Time to update terrains: 468ms 
	Line 2449: DFTFU 1.9.1: Time to update terrains: 481ms 
	Line 2461: DFTFU 1.9.1: Time to update terrains: 545ms 
	Line 2482: DFTFU 1.9.1: Time to update terrains: 516ms 
	Line 2494: DFTFU 1.9.1: Time to update terrains: 537ms 
	Line 2509: DFTFU 1.9.1: Time to update terrains: 589ms 
	Line 2569: DFTFU 1.9.1: Time to update terrains: 526ms 
	Line 2584: DFTFU 1.9.1: Time to update terrains: 582ms 
	Line 2596: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 2611: DFTFU 1.9.1: Time to update terrains: 568ms 
	Line 2626: DFTFU 1.9.1: Time to update terrains: 526ms 
	Line 2638: DFTFU 1.9.1: Time to update terrains: 553ms 
	Line 2665: DFTFU 1.9.1: Time to update terrains: 583ms 
	Line 2677: DFTFU 1.9.1: Time to update terrains: 585ms 
	Line 2698: DFTFU 1.9.1: Time to update terrains: 556ms 
	Line 2713: DFTFU 1.9.1: Time to update terrains: 539ms 
	Line 2725: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 2737: DFTFU 1.9.1: Time to update terrains: 532ms 
	Line 2770: DFTFU 1.9.1: Time to update terrains: 584ms 
	Line 2782: DFTFU 1.9.1: Time to update terrains: 553ms 
	Line 2800: DFTFU 1.9.1: Time to update terrains: 597ms 
	Line 2815: DFTFU 1.9.1: Time to update terrains: 516ms 
	Line 2827: DFTFU 1.9.1: Time to update terrains: 580ms 
	Line 2854: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 2869: DFTFU 1.9.1: Time to update terrains: 582ms 
	Line 2887: DFTFU 1.9.1: Time to update terrains: 535ms 
	Line 2902: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 2914: DFTFU 1.9.1: Time to update terrains: 523ms 
	Line 2929: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 2944: DFTFU 1.9.1: Time to update terrains: 521ms 
	Line 2973: DFTFU 1.9.1: Time to update terrains: 521ms 
	Line 2985: DFTFU 1.9.1: Time to update terrains: 563ms 
	Line 3000: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 3012: DFTFU 1.9.1: Time to update terrains: 585ms 
	Line 3024: DFTFU 1.9.1: Time to update terrains: 537ms 
	Line 3039: DFTFU 1.9.1: Time to update terrains: 583ms 
	Line 3051: DFTFU 1.9.1: Time to update terrains: 513ms 
	Line 3063: DFTFU 1.9.1: Time to update terrains: 590ms 
	Line 3081: DFTFU 1.9.1: Time to update terrains: 544ms 
	Line 3093: DFTFU 1.9.1: Time to update terrains: 584ms 
	Line 3105: DFTFU 1.9.1: Time to update terrains: 521ms 
	Line 3120: DFTFU 1.9.1: Time to update terrains: 571ms 
	Line 3135: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 3150: DFTFU 1.9.1: Time to update terrains: 588ms 
	Line 3162: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 3174: DFTFU 1.9.1: Time to update terrains: 575ms 
	Line 3186: DFTFU 1.9.1: Time to update terrains: 532ms 
	Line 3201: DFTFU 1.9.1: Time to update terrains: 571ms 
	Line 3213: DFTFU 1.9.1: Time to update terrains: 532ms 
	Line 3225: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 3240: DFTFU 1.9.1: Time to update terrains: 539ms 
	Line 3252: DFTFU 1.9.1: Time to update terrains: 539ms 
	Line 3264: DFTFU 1.9.1: Time to update terrains: 591ms 
	Line 3276: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 3288: DFTFU 1.9.1: Time to update terrains: 576ms 
	Line 3303: DFTFU 1.9.1: Time to update terrains: 606ms 
	Line 3315: DFTFU 1.9.1: Time to update terrains: 617ms 
	Line 3327: DFTFU 1.9.1: Time to update terrains: 588ms 
	Line 3339: DFTFU 1.9.1: Time to update terrains: 580ms 
	Line 3363: DFTFU 1.9.1: Time to update terrains: 574ms 
	Line 3375: DFTFU 1.9.1: Time to update terrains: 576ms 
	Line 3393: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 3408: DFTFU 1.9.1: Time to update terrains: 584ms 
	Line 3420: DFTFU 1.9.1: Time to update terrains: 534ms 
	Line 3432: DFTFU 1.9.1: Time to update terrains: 579ms 
	Line 3444: DFTFU 1.9.1: Time to update terrains: 526ms 
	Line 3456: DFTFU 1.9.1: Time to update terrains: 523ms 
	Line 3468: DFTFU 1.9.1: Time to update terrains: 591ms 
	Line 3480: DFTFU 1.9.1: Time to update terrains: 534ms 
	Line 3492: DFTFU 1.9.1: Time to update terrains: 580ms 
	Line 3504: DFTFU 1.9.1: Time to update terrains: 534ms 
	Line 3519: DFTFU 1.9.1: Time to update terrains: 584ms 
	Line 3531: DFTFU 1.9.1: Time to update terrains: 534ms 
	Line 3543: DFTFU 1.9.1: Time to update terrains: 579ms 
	Line 3555: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 3567: DFTFU 1.9.1: Time to update terrains: 583ms 
	Line 3579: DFTFU 1.9.1: Time to update terrains: 542ms 
	Line 3591: DFTFU 1.9.1: Time to update terrains: 525ms 
	Line 3603: DFTFU 1.9.1: Time to update terrains: 572ms 
	Line 3615: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 3627: DFTFU 1.9.1: Time to update terrains: 568ms 
	Line 3639: DFTFU 1.9.1: Time to update terrains: 528ms 
	Line 3651: DFTFU 1.9.1: Time to update terrains: 573ms 
	Line 3663: DFTFU 1.9.1: Time to update terrains: 534ms 
	Line 3675: DFTFU 1.9.1: Time to update terrains: 551ms 
	Line 3687: DFTFU 1.9.1: Time to update terrains: 509ms 
	Line 3699: DFTFU 1.9.1: Time to update terrains: 600ms 
	Line 3714: DFTFU 1.9.1: Time to update terrains: 582ms 
	Line 3735: DFTFU 1.9.1: Time to update terrains: 523ms 
	Line 3747: DFTFU 1.9.1: Time to update terrains: 584ms 
	Line 3759: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 3771: DFTFU 1.9.1: Time to update terrains: 511ms 
	Line 3783: DFTFU 1.9.1: Time to update terrains: 584ms 
	Line 3795: DFTFU 1.9.1: Time to update terrains: 545ms 
	Line 3807: DFTFU 1.9.1: Time to update terrains: 543ms 
	Line 3822: DFTFU 1.9.1: Time to update terrains: 548ms 
	Line 3834: DFTFU 1.9.1: Time to update terrains: 550ms 
	Line 3935: DFTFU 1.9.1: Time to update terrains: 511ms 
	Line 3953: DFTFU 1.9.1: Time to update terrains: 586ms 
	Line 3965: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 3977: DFTFU 1.9.1: Time to update terrains: 578ms 
	Line 3992: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 4016: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 4028: DFTFU 1.9.1: Time to update terrains: 575ms 
	Line 4046: DFTFU 1.9.1: Time to update terrains: 588ms 
	Line 4058: DFTFU 1.9.1: Time to update terrains: 542ms 
	Line 4073: DFTFU 1.9.1: Time to update terrains: 579ms 
	Line 4085: DFTFU 1.9.1: Time to update terrains: 529ms 
	Line 4097: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 4115: DFTFU 1.9.1: Time to update terrains: 548ms 
	Line 4130: DFTFU 1.9.1: Time to update terrains: 535ms 
	Line 4142: DFTFU 1.9.1: Time to update terrains: 542ms 
	Line 4154: DFTFU 1.9.1: Time to update terrains: 534ms 
	Line 4166: DFTFU 1.9.1: Time to update terrains: 583ms 
	Line 4178: DFTFU 1.9.1: Time to update terrains: 534ms 
	Line 4190: DFTFU 1.9.1: Time to update terrains: 614ms 
	Line 4202: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 4214: DFTFU 1.9.1: Time to update terrains: 581ms 
	Line 4226: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 4238: DFTFU 1.9.1: Time to update terrains: 585ms 
	Line 4253: DFTFU 1.9.1: Time to update terrains: 540ms 
	Line 4268: DFTFU 1.9.1: Time to update terrains: 589ms 
	Line 4283: DFTFU 1.9.1: Time to update terrains: 537ms 
	Line 4295: DFTFU 1.9.1: Time to update terrains: 605ms 
	Line 4316: DFTFU 1.9.1: Time to update terrains: 551ms 
	Line 4328: DFTFU 1.9.1: Time to update terrains: 570ms 
	Line 4340: DFTFU 1.9.1: Time to update terrains: 524ms 
	Line 4352: DFTFU 1.9.1: Time to update terrains: 507ms 
	Line 4364: DFTFU 1.9.1: Time to update terrains: 535ms 
	Line 4376: DFTFU 1.9.1: Time to update terrains: 529ms 
	Line 4391: DFTFU 1.9.1: Time to update terrains: 580ms 
	Line 4403: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 4415: DFTFU 1.9.1: Time to update terrains: 590ms 
	Line 4430: DFTFU 1.9.1: Time to update terrains: 547ms 
	Line 4442: DFTFU 1.9.1: Time to update terrains: 566ms 
	Line 4454: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 4469: DFTFU 1.9.1: Time to update terrains: 612ms 
	Line 4481: DFTFU 1.9.1: Time to update terrains: 518ms 
	Line 4496: DFTFU 1.9.1: Time to update terrains: 574ms 
	Line 4508: DFTFU 1.9.1: Time to update terrains: 520ms 
	Line 4523: DFTFU 1.9.1: Time to update terrains: 529ms 
	Line 4535: DFTFU 1.9.1: Time to update terrains: 579ms 
	Line 4547: DFTFU 1.9.1: Time to update terrains: 515ms 
	Line 4562: DFTFU 1.9.1: Time to update terrains: 561ms 
	Line 4574: DFTFU 1.9.1: Time to update terrains: 521ms 
	Line 4586: DFTFU 1.9.1: Time to update terrains: 576ms 
	Line 4598: DFTFU 1.9.1: Time to update terrains: 519ms 
	Line 4639: DFTFU 1.9.1: Time to update terrains: 539ms 
	Line 4651: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 4663: DFTFU 1.9.1: Time to update terrains: 539ms 
	Line 4678: DFTFU 1.9.1: Time to update terrains: 538ms 
	Line 4693: DFTFU 1.9.1: Time to update terrains: 510ms 
	Line 4717: DFTFU 1.9.1: Time to update terrains: 523ms 
	Line 4729: DFTFU 1.9.1: Time to update terrains: 519ms 
	Line 4741: DFTFU 1.9.1: Time to update terrains: 575ms 
	Line 4753: DFTFU 1.9.1: Time to update terrains: 513ms 
	Line 4765: DFTFU 1.9.1: Time to update terrains: 517ms 
	Line 4777: DFTFU 1.9.1: Time to update terrains: 569ms 
	Line 4792: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 4804: DFTFU 1.9.1: Time to update terrains: 529ms 
	Line 4816: DFTFU 1.9.1: Time to update terrains: 569ms 
	Line 4831: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 4846: DFTFU 1.9.1: Time to update terrains: 531ms 
	Line 4861: DFTFU 1.9.1: Time to update terrains: 575ms 
	Line 4873: DFTFU 1.9.1: Time to update terrains: 548ms 
	Line 4888: DFTFU 1.9.1: Time to update terrains: 585ms 
	Line 4906: DFTFU 1.9.1: Time to update terrains: 521ms 
	Line 4924: DFTFU 1.9.1: Time to update terrains: 542ms 
	Line 4936: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 4951: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 4966: DFTFU 1.9.1: Time to update terrains: 597ms 
	Line 4981: DFTFU 1.9.1: Time to update terrains: 536ms 
	Line 4993: DFTFU 1.9.1: Time to update terrains: 580ms 
	Line 5008: DFTFU 1.9.1: Time to update terrains: 525ms 
	Line 5020: DFTFU 1.9.1: Time to update terrains: 586ms 
	Line 5032: DFTFU 1.9.1: Time to update terrains: 527ms 
	Line 5044: DFTFU 1.9.1: Time to update terrains: 520ms 
	Line 5056: DFTFU 1.9.1: Time to update terrains: 577ms 
	Line 5068: DFTFU 1.9.1: Time to update terrains: 523ms 
	Line 5086: DFTFU 1.9.1: Time to update terrains: 580ms 
	Line 5098: DFTFU 1.9.1: Time to update terrains: 533ms 
	Line 5113: DFTFU 1.9.1: Time to update terrains: 577ms 
	Line 5125: DFTFU 1.9.1: Time to update terrains: 530ms 
	Line 5137: DFTFU 1.9.1: Time to update terrains: 522ms 
	Line 5149: DFTFU 1.9.1: Time to update terrains: 575ms 
	Line 5161: DFTFU 1.9.1: Time to update terrains: 543ms 
	Line 5176: DFTFU 1.9.1: Time to update terrains: 605ms 
	Line 5194: DFTFU 1.9.1: Time to update terrains: 552ms 
	Line 5206: DFTFU 1.9.1: Time to update terrains: 553ms 
	Line 5218: DFTFU 1.9.1: Time to update terrains: 570ms 
	Line 5230: DFTFU 1.9.1: Time to update terrains: 577ms 
	Line 5242: DFTFU 1.9.1: Time to update terrains: 542ms 

As I mentioned in my first post, those batches are still running because I'm not sure how to prevent that and just set the values directly from the array I read from the file cache. I imagine that will see a bigger speed increase.

Nevertheless I'm impressed with how well your version runs compared to the Tedious Travel prototype, I don't have the most powerful rig but I survived that journey at 40x speed without needing the cache. It was a bit choppier in places.

And yeah, Pango's pole-and-fence example made the 'why' click, I thought it had something to do with the number of batches but I was probably confusing that with something else.

In any case, I'm not done with this yet I'm pretty sure I'll see a bigger increase once I prevent that batch job from happening.

[BadLuckBurt]

I'm not sure this will make much difference to the terrain performance really, as it's multi-threaded and only one part of the chain. The parts that can't be parallelised are when creating unity game objects as that has to be done on the main thread, creating nature billboards and locations etc. It may have utility for anyone with plenty of ram but few processor cores though...

I did a test by disabling the nature billboards code and that shaves off about 100ms. So you are absolutely right, the terrain generation is not a bottleneck at all. In fact, it's a freaking beast I even disabled the reading from the Woods file and the performance gain was just a few milliseconds.

For the little performance gain this offers, it's probably not worth the generation time and disk space. It wouldn't even work for samplers with a higher resolution as 129x129 samples would already mean 4 times the data, weighing in at 100GB.

I've done that 40x trip about 5 times now and haven't had a single crash. I will try again tomorrow with a lower TerrainDistance, maybe it's not crashing because it's already loaded a lot of tiles ahead of time, preventing the overload you mentioned.