Integrating terrain texturing with basic roads texturing

Discuss modding questions and implementation details.
User avatar
Hazelnut
Posts: 3015
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Integrating terrain texturing with basic roads texturing

Post by Hazelnut »

I managed to get Interesting Terrains working with roads very quickly tonight, as you can see from this picture:
IT-Roads.JPG
IT-Roads.JPG (171.16 KiB) Viewed 1706 times

Can't really take the credit for this though... Monobelisk did the work back last November and provided a way for the terrain tile data that his mod is creating during the terrain height sampler to be accessed by my terrain texturer in basic roads. They even had put a comment with the mod message callback code for me. :ugeek: Now I feel really bad I didn't look at this at the end of last year. :(

Idea #1:

Generate the terrain tile data and provide them to roads mod. So this is basically what interesting terrain is doing, which is pretty easy for it to do since it's actually generating this data during the height sampling so all that my BR mod needs to do is not schedule the standard default tile data generation job and instead simply request the already generated tile data for the map pixel from the IT mod, then use that as the input to my BR job.

In theory this could also be done if a mod (for example the Wilderness Overhaul by Daniel87) schedules a job to generate tile data, as long as the standard marching algorithm in DFU (the second of the texturing jobs) is still used as it's in this job that my BR road painting code lives. Achieving this would probably require a way to contribute a version of the first job from one mod to another maybe using delegates. Or maybe splitting the ITerrainTexturing interface somehow, though I'd prefer to avoid the latter.


Idea #2:

Possibly allow mods to use the road layout code without just copying it. (not that I mind per se, but it would cause a maintenance overhead) The easiest way would be to provide access to the road painting code that could be scheduled as a separate post processing job. The issue with this is my code relies on the output terrain texture data having the location tiles but nothing else so it doesn't override the location tiles with roads. Possibly better if I provided a roads painting job that only did that and could be scheduled in between the tile data generation and the tiling jobs. Not sure how to achieve this due to namespaces and code compilation issues. (normally not an issue since the mod message uses a callback system, but using this for each tile in the job would cripple performance even if it worked I suspect)



What I really need now is to look through the code for Daniel87's Wilderness Overhaul mod and find the answer to which job(s) of the texturing it overrides and roughly what the new jobs are doing. A task for another day...
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Hazelnut
Posts: 3015
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Integrating terrain texturing with basic roads texturing

Post by Hazelnut »

Having now grabbed the code for Wilderness Overhaul (WO) I definitely need to go for idea 2 since it has custom tile generation and assignment code. So I have restructured the code for BR and provide a way for other mods to schedule the paint roads job (& smooth too if option enabled) in between tile generation and assignment. This means that BR on it's own will be slightly less efficient since it's no longer combining road painting with tile assignment, but we will see if that actually makes any material difference on potato pc's before I worry about re-adding that.

Next stage is to modify WO code to schedule the roads jobs. Got it working in the Unity editor, main issue is that the natures are being placed on road tiles as you can see below:
WO-Roads.JPG
WO-Roads.JPG (173.51 KiB) Viewed 1647 times

I tried having the nature continue for any tile except plain grass, dirt, or stone like the default DFU one but that made no difference. Daniel87 I think we'll need to discuss this and come up with a solution once everything else is working well.

Next step I tried to test this with built mod packages, to test the intra-mod communication still works but unfortunately I ran into some issues:
  • Had to remove two files from the dfmod file since they are not in my fork of the repo. (WOTilemapTerrainMaterialProvider.cs and TEXTURE.305.asset) I assume these are files removed since the last time you built the mod.
  • Compilation error when the mod code is compiled with the runtime mcs or when I try to build with pre-compiled option.

    Code: Select all

    Error (122): `DaggerfallWorkshop.TileTexArrUniforms' is inaccessible due to its protection level
    Error (): 6(39,30): DaggerfallWorkshop.WOTilemapTextureArrayTerrainMaterialProvider.PromoteMaterial(DaggerfallWorkshop.DaggerfallTerrain, DaggerfallWorkshop.TerrainMaterialData)
Just in case, since I was running with the PR2007 branch and it was merged last night, I switched to the latest master DFU code. Still encountered the same compilation issue. Daniel87 I think you need to take a look at this ASAP. Hopefully it will just need TheLacus to make a protection change to follow on from PR2007, but you will need this fixed to be able to release the mod.
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Daniel87
Posts: 391
Joined: Thu Nov 28, 2019 6:25 pm

Re: Integrating terrain texturing with basic roads texturing

Post by Daniel87 »

Hazelnut wrote: Thu May 20, 2021 11:36 am Having now grabbed the code for Wilderness Overhaul (WO) I definitely need to go for idea 2 since it has custom tile generation and assignment code. So I have restructured the code for BR and provide a way for other mods to schedule the paint roads job (& smooth too if option enabled) in between tile generation and assignment. This means that BR on it's own will be slightly less efficient since it's no longer combining road painting with tile assignment, but we will see if that actually makes any material difference on potato pc's before I worry about re-adding that.

Next stage is to modify WO code to schedule the roads jobs. Got it working in the Unity editor, main issue is that the natures are being placed on road tiles as you can see below:

WO-Roads.JPG


I tried having the nature continue for any tile except plain grass, dirt, or stone like the default DFU one but that made no difference. Daniel87 I think we'll need to discuss this and come up with a solution once everything else is working well.

Next step I tried to test this with built mod packages, to test the intra-mod communication still works but unfortunately I ran into some issues:
  • Had to remove two files from the dfmod file since they are not in my fork of the repo. (WOTilemapTerrainMaterialProvider.cs and TEXTURE.305.asset) I assume these are files removed since the last time you built the mod.
  • Compilation error when the mod code is compiled with the runtime mcs or when I try to build with pre-compiled option.

    Code: Select all

    Error (122): `DaggerfallWorkshop.TileTexArrUniforms' is inaccessible due to its protection level
    Error (): 6(39,30): DaggerfallWorkshop.WOTilemapTextureArrayTerrainMaterialProvider.PromoteMaterial(DaggerfallWorkshop.DaggerfallTerrain, DaggerfallWorkshop.TerrainMaterialData)
Just in case, since I was running with the PR2007 branch and it was merged last night, I switched to the latest master DFU code. Still encountered the same compilation issue. Daniel87 I think you need to take a look at this ASAP. Hopefully it will just need TheLacus to make a protection change to follow on from PR2007, but you will need this fixed to be able to release the mod.
I believe I had a solution for the placement of nature on road tiles, but that was before I changed the tile texturing algo, so possibly it doesn't respond to it anymore (but it should, as I only appended textures to the array rather than shuffling their indices). Possibly I commented it out as there was no support for BR for the time being anyway.

There is one possible problem with my terrain texturing that we might crash into: I basically added 4 (or 5?) new textures to the array to completely fill it up to 64 (absolute hard cap because of the ridiculous byte structure of ol' DF). I now realize, that there is no texture that combines road + rock. There is only road + dirt and road + grass. So either I will have to create some strict rules when tiling terrain to absolutely avoid texturing rock where later on road is being placed or I will have to completely rethink the way how I texture the terrain. So far I couldn't come up with anything fruitful, as there is no possible way around the byte structure encoding flip, rotation and index of the texture. This will always limit us to 64 possible textures, no matter how we twist or turn it.

This very problem already prohibited me from creating water + rock tiles and limited the way the terrain can be textured.

EDIT: The code that checks for road tiles and then avoids planting plants on it is still in there in my code, but I forgot to change the texture IDs after I completely changed the texturing system and created new Texture2DArrays. I just pushed the fix to GitHub. Had to add different texture IDs into the TileTypeCheck function. Should work now (couldn't test it yet.)

I sadly have 0 experience with Unitys Job System and only rudimentarily understand how it works so far.
So the basic load order would be IET (for heightmaps) -> WO (for Tile texturing) -> Basic Roads (for Road texture overlay)?

EDIT2:
I just deleted my whole Daggerfall Unity folder and re-downloaded the current master from Interkarma. I will give it 10 minutes now and then HAVE to head to bed, as I got up 6am and now it's 1am. I will invest the whole day tomorrow to try and fix all bugs, so we can get started making them compatible.

God, I HATE GitHub from the depth of my soul and wish for it to burn eternally in hell.
Image
I tried and failed so often to get nicely along with this platform and have given up long ago. This is what I get, even after completely deleting the DFU project and redownloading it. No idea why I am still in my own crap fork instead of the original one or if I am supposed to be in there. I have no bloody idea how this crap works. I will re-download every time there is a new DFU update, rather than wasting 10 more hours of my lifetime into figuring out this crappy platform with it's convoluted system.

EDIT3:
I tried to build my mod but DFU doesn't find it... Is there any comprehensive tutorial on how to build the mod?
What is the version field in "Mod Settings" supposed to be? Mod version? DFU version?

Anyway, enough hate. Will have to sleep now and deal with this tomorrow.
Last edited by Daniel87 on Thu May 20, 2021 5:51 pm, edited 1 time in total.
In Julianos we Trust.

User avatar
Hazelnut
Posts: 3015
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Integrating terrain texturing with basic roads texturing

Post by Hazelnut »

Thanks for the response, will reply to each part later. Just for now though, the only slightly urgent issue is the compilation problems that might need core DFU changes to go in before next release which is coming up, otherwise you'd have to wait for the following one to release your mod with these updates.

So yeah, go get some sleep! :) If you can look at that tomorrow great, but the roads stuff is not so urgent so we'll do it as we can.

EDIT: I feel your pain, can easily get into a mess with GitHub. I can help you out if you need it.
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Daniel87
Posts: 391
Joined: Thu Nov 28, 2019 6:25 pm

Re: Integrating terrain texturing with basic roads texturing

Post by Daniel87 »

Hazelnut wrote: Thu May 20, 2021 5:33 pm EDIT: I feel your pain, can easily get into a mess with GitHub. I can help you out if you need it.
That would be awesome. All I tried was following a tutorial to push a commit to Interkarma for a small bug I found. I ended up giving up after messing arround too much and detaching my head as well as the repositories for too often. I told Interkarma about the bug in a PM and he already fixed it but since then I am stuck in GitHub and can't pull the current master anymore.

So I got myself a clean download of 0.11.3 now. I then changed that version number in my mod and mod settings and got the same error. Seems like it has to do with the Shader and PromoteMaterial() function. I was so glad this ran in the Editor, as I had no idea what I was doing, but sadly it seems that's the only place where it ran. It's a function I had to borrow from MaterialReader.cs, to get it working. Actually I believe it is already obsolete. It stems from a different approach I made for terrain texturing but it got scrapped once I created my own texture2Darrays based on the original arrays with added textures.

LAST EDIT: Okay, the files could safely be deleted and were from an obsolete approach. Now I don't get errors when building but still can't find my own mod after building. No idea why that happens. Can you confirm the same problem? (You'll have to pull the newest version).

"Unity Versions - mods created in newer versions of unity might not be compatible with older versions of unity. This is noticeable if the mod is in the mod directory, but doesn't show up in the list of mods." this is all I could find about the issue I am facing. But I could swear that I entered 0.11.3 in the mod settings header and in the mod builder
In Julianos we Trust.

User avatar
Hazelnut
Posts: 3015
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Integrating terrain texturing with basic roads texturing

Post by Hazelnut »

So good news, grabbed the latest code from you and the mod compiles just fine now. I have completed the roads integration and it works well as far as I can see. Have sent a PR for the changes over to you. Hopefully they're acceptable. To test it you will need to use the new test build of Basic Roads attached to this post. I tested dfmod builds, but only in the unity editor. I do still need to do a final check with an actual DFU release, but I don't expect any issues.

I would suggest adding an optional dependency on BasicRoads v1.0.0 in your mod config before you release so people are notified if they try to use your new build with an older version of roads. Once we've worked through any issues I'll upload the new v1.0 of roads to nexus.

The natures are still being placed on roads. The default nature layout code will only place natures on plain (full) grass, dirt, or stone tiles and hence any road tiles added by my mod are ignored. Unfortunately I don't understand your code well enough to understand why this is happening.

Regarding the stone tiles, there has never been a tile that combines them so for this reason my roads that go over stone diagonally use the half road/grass tiles. It doesn't look great, but it's quite rare and the only other option was to use full stone and that looked even worse. I would leave it for now, since it's the same regardless of your mod. Potentially we can work to "create some strict rules when tiling terrain to absolutely avoid texturing rock where later on road is being placed" but I would not say this is a priority for now at least. (considering you're busy)

One other thing I did notice is that your assign tiles job is quite computationally expensive with a lot of memory allocations. I can see that the code is still being written as there's lots of commented out sections etc. but once this is more final, I'll try to help see if anything can be done to alleviate this. Just recording it here for now because I'll probably forget all about it.

Lastly I just want to say that the nature layouts look great from the little I've seen while testing roads with your mod! Great stuff, and very well worth the time to re-factor my code to be compatible.

Although I already had a working integration for Interesting Terrain (idea #1) I've scrapped it and gone with the same style of integration as I used with yours for consistency.
Attachments
basicroads-0.9.1.zip
(80.72 KiB) Downloaded 59 times
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Daniel87
Posts: 391
Joined: Thu Nov 28, 2019 6:25 pm

Re: Integrating terrain texturing with basic roads texturing

Post by Daniel87 »

Hazelnut wrote: Thu May 20, 2021 9:51 pm So good news, grabbed the latest code from you and the mod compiles just fine now. I have completed the roads integration and it works well as far as I can see. Have sent a PR for the changes over to you. Hopefully they're acceptable. To test it you will need to use the new test build of Basic Roads attached to this post. I tested dfmod builds, but only in the unity editor. I do still need to do a final check with an actual DFU release, but I don't expect any issues.

I would suggest adding an optional dependency on BasicRoads v1.0.0 in your mod config before you release so people are notified if they try to use your new build with an older version of roads. Once we've worked through any issues I'll upload the new v1.0 of roads to nexus.

The natures are still being placed on roads. The default nature layout code will only place natures on plain (full) grass, dirt, or stone tiles and hence any road tiles added by my mod are ignored. Unfortunately I don't understand your code well enough to understand why this is happening.

Regarding the stone tiles, there has never been a tile that combines them so for this reason my roads that go over stone diagonally use the half road/grass tiles. It doesn't look great, but it's quite rare and the only other option was to use full stone and that looked even worse. I would leave it for now, since it's the same regardless of your mod. Potentially we can work to "create some strict rules when tiling terrain to absolutely avoid texturing rock where later on road is being placed" but I would not say this is a priority for now at least. (considering you're busy)

One other thing I did notice is that your assign tiles job is quite computationally expensive with a lot of memory allocations. I can see that the code is still being written as there's lots of commented out sections etc. but once this is more final, I'll try to help see if anything can be done to alleviate this. Just recording it here for now because I'll probably forget all about it.

Lastly I just want to say that the nature layouts look great from the little I've seen while testing roads with your mod! Great stuff, and very well worth the time to re-factor my code to be compatible.

Although I already had a working integration for Interesting Terrain (idea #1) I've scrapped it and gone with the same style of integration as I used with yours for consistency.
Thank you for your Pull Request.
I merged it with my code now and also fixed the nature placement on roads issue.

My major problem right now is: I fail to find my own Mod outside of the Unity Editor in the actual game when building and testing in the actual game. Can't find it under mods in DFU 0.11.3. I am using the Linux build and built the mod for Linux as well (with the pre-compiled option and without, both seem not to work for me)

I read in the tutorial this might be due to differing versions, but I made sure both mod settings and mod builder have the Version 0.1 set for my mod and 0.11.3 for DFU, which is the DFU version I am running (downloaded it yesterday).

Were you successfully building my mod and able to see it outside of the editor when testing with yours?
Also: How do I set up dependencies for Mods in the Mod Builder Interface? (I would like to make my mod depending on IEA and BR)

APPENDIX:
I also just came across a small incompatibility between your mod and the Real Grass 2 mod.
Real Grass 2 distributes grass sprites in a similar fashion as my mod does. This results in placement of grass on top of paths, due to the way how BR utilizes the tile textures, which are considered "border textures between dirt and grass" for Real Grass 2.

Image

Btw, wait until you see the fireflies appearing in the late twilight hours and the shooting stars cross the sky <3
Should already be implemented in the moderate woodland climate.
In Julianos we Trust.

User avatar
Hazelnut
Posts: 3015
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Integrating terrain texturing with basic roads texturing

Post by Hazelnut »

First the good news.. roads are now clear of vegetation! :D looks great.

To add dependencies, click the 'Open Dependencies Editor' button on the mod builder window. Then add basicroads, is compat but not required, must be loaded before and tick version and put a 1 into the first input field. (this will cause warnings until I actually release BR v1.0 though, but you can just click yes to continue for now) Don't forget to save the settings :)

Secondly, the less good news. I can successfully run your mod in the current DFU release 0.11.3 after building it myself. This is on windows 10 without pre-compile turned on. However it does cause exceptions to be thrown for me. No idea why you can't see it in the mod list at all though, my only guess is maybe you pre-compiled and the exceptions prevent the mod even loading. (whereas I get to runtime before it breaks)

I've attached my latest build of your mod to this post so you can try it, though it is windows build so not sure if that will work. (note it also includes the additional if statement I added below) Make sure you disable flys/stars unless you want the exceptions and a black screen.

When running in Unity editor but with the built dfmods in SA/Mods folder (rather than virtual) I get NullReferenceExceptions if either fireflys or shooting stars are enabled in mod settings. When running in the released 0.11.3 DFU build I get the errors below if either are enabled.

Note that I had to add this if statement for it to work with stars disabled:

Code: Select all

// Adds one shooting star Particle System of every MapPixel
if (shootingStarsExist)
    AddShootingStar(dfTerrain, dfBillboardBatch, 90f, 900f, shootingStarsMinimum, shootingStarsMaximum); // Shooting Stars


Errors running in release with flys/stars enabled:

Code: Select all

WARNING: Shader Unsupported: 'Hidden/Nature/Terrain/Utilities' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
ERROR: Shader Hidden/Nature/Terrain/Utilities shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ArgumentException: The Object you want to instantiate is null.
  at UnityEngine.Object.Instantiate (UnityEngine.Object original, UnityEngine.Vector3 position, UnityEngine.Quaternion rotation, UnityEngine.Transform parent) [0x00048] in <548b4fa0e7e04f27a1b7580930bfb7dc>:0
  at UnityEngine.Object.Instantiate[T] (T original, UnityEngine.Vector3 position, UnityEngine.Quaternion rotation, UnityEngine.Transform parent) [0x00001] in <548b4fa0e7e04f27a1b7580930bfb7dc>:0
  at WildernessOverhaul.WOTerrainNature.AddShootingStar (DaggerfallWorkshop.DaggerfallTerrain dfTerrain, DaggerfallWorkshop.DaggerfallBillboardBatch dfBillboardBatch, System.Single rotationAngleX, System.Single heightInTheSky, System.Single sSMin, System.Single sSMax) [0x00060] in <4ded227c3dec4d169bf4336daee09bea>:0
  at WildernessOverhaul.WOTerrainNature.LayoutNature (DaggerfallWorkshop.DaggerfallTerrain dfTerrain, DaggerfallWorkshop.DaggerfallBillboardBatch dfBillboardBatch, System.Single terrainScale, System.Int32 terrainDist) [0x00356] in <4ded227c3dec4d169bf4336daee09bea>:0
  at DaggerfallWorkshop.StreamingWorld.UpdateTerrainNature (DaggerfallWorkshop.StreamingWorld+TerrainDesc terrainDesc) [0x00089] in <57ff85fdde09455d906d701e17b0a6ad>:0
  at DaggerfallWorkshop.StreamingWorld.InitPlayerTerrain () [0x00057] in <57ff85fdde09455d906d701e17b0a6ad>:0
  at DaggerfallWorkshop.StreamingWorld.Update () [0x00093] in <57ff85fdde09455d906d701e17b0a6ad>:0

(Filename: <548b4fa0e7e04f27a1b7580930bfb7dc> Line: 0)

Entering new map pixel X=193, Y=208
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

ArgumentException: The Object you want to instantiate is null.
  at UnityEngine.Object.Instantiate (UnityEngine.Object original, UnityEngine.Vector3 position, UnityEngine.Quaternion rotation, UnityEngine.Transform parent) [0x00048] in <548b4fa0e7e04f27a1b7580930bfb7dc>:0
  at UnityEngine.Object.Instantiate[T] (T original, UnityEngine.Vector3 position, UnityEngine.Quaternion rotation, UnityEngine.Transform parent) [0x00001] in <548b4fa0e7e04f27a1b7580930bfb7dc>:0
  at WildernessOverhaul.WOTerrainNature.AddShootingStar (DaggerfallWorkshop.DaggerfallTerrain dfTerrain, DaggerfallWorkshop.DaggerfallBillboardBatch dfBillboardBatch, System.Single rotationAngleX, System.Single heightInTheSky, System.Single sSMin, System.Single sSMax) [0x00060] in <4ded227c3dec4d169bf4336daee09bea>:0
  at WildernessOverhaul.WOTerrainNature.LayoutNature (DaggerfallWorkshop.DaggerfallTerrain dfTerrain, DaggerfallWorkshop.DaggerfallBillboardBatch dfBillboardBatch, System.Single terrainScale, System.Int32 terrainDist) [0x00356] in <4ded227c3dec4d169bf4336daee09bea>:0
  at DaggerfallWorkshop.StreamingWorld.UpdateTerrainNature (DaggerfallWorkshop.StreamingWorld+TerrainDesc terrainDesc) [0x00089] in <57ff85fdde09455d906d701e17b0a6ad>:0
  at DaggerfallWorkshop.StreamingWorld.InitPlayerTerrain () [0x00057] in <57ff85fdde09455d906d701e17b0a6ad>:0
  at DaggerfallWorkshop.StreamingWorld.Update () [0x00093] in <57ff85fdde09455d906d701e17b0a6ad>:0
Attachments
WO-hazelnut.zip
(1.76 MiB) Downloaded 64 times
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Daniel87
Posts: 391
Joined: Thu Nov 28, 2019 6:25 pm

Re: Integrating terrain texturing with basic roads texturing

Post by Daniel87 »

Interesting, your build seems to work but as you said, only when deactivating fireflies and shooting stars.

Those are actually made up by prefabs and placed in a Resources folder. This way it worked in the editor but I am not sure if this is the right way for doing it when the mod is built.
Do you know where I should place things like Prefabs, textures, etc. ?

Also interesting is, that your compiled version of my mod is 2.1MB, while mine is 3MB. Did you leave out any files?
Last edited by Daniel87 on Fri May 21, 2021 11:32 am, edited 1 time in total.
In Julianos we Trust.

User avatar
Hazelnut
Posts: 3015
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Integrating terrain texturing with basic roads texturing

Post by Hazelnut »

FYI Just tried runnng the mod files attached in this thread on Linux 0.11.3 DFU with my laptop and they work fine with the flys and stars disabled in settings.
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

Post Reply