So, I'm kind of speaking in ignorance here, but from looking more closely at how the terrain flats are rendered in DFU, it appears as if all the billboards in a location are combined into one mesh. I assume since they all share the same material and texture (atlas), this allows them to be rendered in one call without the CPU overhead of dynamic batching. Super efficient. Since we are dealing with LODs and dynamically enabling/disabling renderers in the process, I don't think we can take advantage of the same technique. But even dynamic batching (as apposed to static batching) we should have been getting better performance than what we're currently getting with the Speedtree billboards... so I don't think batching is the problem here - I still believe it is either culling calculations or the crappy Speedtree billboards like you said.
I think your idea about creating a new atlased billboard shader would work just fine. All the billboards would share the same material, so they should dynamically batch into one draw call. I'm pretty bad with shaders, but I think I may be able to copy most of "DaggerfallBillboardBatch.shader" into a new shader that supports texture atlases so we can just reference an index in that atlas for each billboard.
At this point, I want to try one last thing to get Daggerfall Billboard Batched script to work, but I think ultimately we will have to do as you suggest and make our own simple shader and texture atlas. If it comes to that, I can repurpose my Spriter tool to bake albedo and normal textures of the 3D models, then pack them into an atlas. But first, I'll just start with the vanilla textures.
Can you upload a package with the entire set of temperate forest models? That will give me a good start on benchmarking once I get something working.Ok, let me know if you'll need anything.