Creating Custom Worlds

Post here if you need help getting started with Daggerfall Unity or just want to clarify a potential bug. Questions about playing or modding classic Daggerfall should be posted to Community.
Post Reply
odival
Posts: 4
Joined: Fri Dec 04, 2015 9:27 pm

Creating Custom Worlds

Post by odival » Fri Dec 04, 2015 9:38 pm

Hello, I came across DFTFU recently, and the potential is just amazing! I followed the basic tutorials and setup the city and dungeon quite painlessly. However, I now want to try to create a custom world, using heightmaps. I actually already created a huge continent in World Machine, and divided in several terrains in Unity, it's all working nicelly.
But how can I change DFTFU to use this terrain instead of the original one? I want the Tools to work with my terrain just like they would the original games, in a way that the atlases, floating origin and even the mods we find here (like the Real Grass one) would work on my world as well. Is this doable?
I'm a begginer scripter but if you guys could point me in the right direction I think I can pull this off. So where do I begin? :mrgreen:

User avatar
Interkarma
Posts: 4863
Joined: Sun Mar 22, 2015 1:51 am

Re: Creating Custom Worlds

Post by Interkarma » Fri Dec 04, 2015 11:46 pm

G'day odival, welcome to the forums. :)

It's possible to feed in custom heightmap data, even higher resolution heightmap data. Before I cover that, there are several limitations to be aware of right now though.
  1. The world is still built procedurally by StreamingWorld at runtime. This means you can provide custom heightmap data to StreamingWorld through scripting, but not custom hand-textured terrains tiles created in Unity. With 1000x500 terrains of 832.3072m2 (32768 square inches) each, the manual approach isn't really practical anyway.
  2. StreamingWorld will still try to place locations at their same map pixel coordinates. This means your new terrain heightmap data will need to follow the crescent shape of the Illiac Bay at least closely enough that locations don't end up in the water. At this time, there's no way to override the location placement (although it's certainly possible if you want to hack around in StreamingWorld a little).
  3. Terrain texturing still uses the retro tilemap texturing seen on the default terrain. It's currently not possible to change climate map or political maps.
  4. Terrain streaming works best with lower resolution heightmaps. The more data it needs to work with, the more noticeable the pause when new terrain tiles are placed.
So basically you can change the heightmap data that StreamingWorld uses to create the world at runtime, but not much else. This still has some benefits however. For example, you could make a higher resolution version of the Illiac Bay, or a more detailed procedural generation method as Nystul did for his far terrain mod.

So how do we go about injecting our own custom heightmap data? This is done by creating a new implementation of ITerrainSampler, an interface which provides height samples to StreamingWorld through a defined set of parameters and methods.

When constructing terrains at runtime, StreamingWorld will ask the currently registered TerrainSampler "what are the height samples for this map tile?" and TerrainSampler can return those samples based on any source. This could be a large heightmap created in WorldMachine, a fully procedural method, or something crazy like USGS data of the planet Mars.

There are already a few example implementations of ITerrainSampler that should help you get started creating your own. You'll find these in the Scripts/Terrain folder. Here's a short description of each.
  • SimpleTerrainSampler. The simplest possible sampler. Generates a completely flat terrain.
  • NoiseTerrainSampler. Generates continuous terrain from Perlin Noise.
  • DefaultTerrainSampler. Generates terrain from Daggerfall's own heightmap data.
To test any of these samplers, open DaggerfallUnity.cs in the Scripts folder and change the following line to use another sampler, such as new SimpleTerrainSampler().

Code: Select all

ITerrainSampler terrainSampler = new DefaultTerrainSampler();
To use your own world, you must implement ITerrainSampler for your custom heightmap data. The world is basically 1000x500 tiles of at least 129x129 samples each. This means the minimum size world is 129,000 x 64,500 samples just to hit the same detail as the retro terrain. The number of samples per tile can be 257, 513, 1025, etc. but more detail per tile means more memory and more processing time.

I hope that at least gets you started. Please feel free to ask any questions and I will do what I can to help. Good luck! :)

odival
Posts: 4
Joined: Fri Dec 04, 2015 9:27 pm

Re: Creating Custom Worlds

Post by odival » Sat Dec 05, 2015 1:16 am

Thanks for taking the time to give such a insightfull reply. I still have some doubts though and I'm sorry in advance for anoying you with such noobish questions:

1) You said the world is made of 1000 x 500 terrains, each one of these having 832m², right? Are those values fixed, or can I have a world made of, eg., 1000x600 terrains of 3.500m², or 500x500 terrains of 500m²? If it's not fixed, where should I change these values?

2) Are political and climate maps going to be customizable in future versions?

Thanks a lot for helping me out

User avatar
Interkarma
Posts: 4863
Joined: Sun Mar 22, 2015 1:51 am

Re: Creating Custom Worlds

Post by Interkarma » Sat Dec 05, 2015 2:06 am

I'd like to allow for runtime modification of the climate and political maps at a future time. You could already do this now with a little hacking in the core scripts but there's no plug-in interface like ITerrainSampler for this currently.

The terrain dimension is fixed to the same units that Daggerfall uses (which is 32768x32768 inches per tile). This is the dimension of a full-size city tile (e.g. Daggerfall, Wayrest) plus a bit of space around the outside of city walls.

The number of terrain tiles (1000x500 for game world) are also fixed. These are the dimensions Daggerfall uses for all of its maps, such as the base height map, climate map, and political map. There is a very tight coupling between all these systems to accurately stream Daggerfall's world at runtime. For example, all the overland locations in MAPS.BSA are defined to exist within a world with those precise dimensions. You can't change one set of parameters without effecting everything else.

Another way of looking at things is that DFTFU is designed to work with Daggerfall's file formats. You can use those files in one of two ways:
  1. Import content like cities and dungeons directly to create smaller custom game worlds. This is how I created the Direnni Tower demo. You could build and script smaller worlds any way you like here.
  2. Work within Daggerfall's constraints and inject procedural content designed to replace or augment Daggerfall's content. You are however limited to Daggerfall's "style" of doing things.
You need to keep in mind that Daggerfall's world is immense. Even the lowest resolution height map of 129k * 64.5k samples (that's almost an 8GB heightmap!) isn't really practical to create by hand or even using an external tool. Simply doubling the heightmap resolution leads to a ~32GB heightmap. That's why you see me using techniques like bicubic interpolation overlaid with noise to stretch a very small amount of data into a more detailed state. This blog article has a little more about the process.

To avoid these mad storage numbers requires a great deal of data coherence and procedural generation, which is exactly how Daggerfall rolls. It's one of the reasons Daggerfall appeals to coders, it was easily one of the most sophisticated procedurally generated worlds of its time and is still fascinating today in how its systems work. I think Julian LeFay is quite an under-appreciated genius.

If you can tell me a little more about what you want to accomplish, I might be able to offer some guidance on how to bend the tools down that line. :)

odival
Posts: 4
Joined: Fri Dec 04, 2015 9:27 pm

Re: Creating Custom Worlds

Post by odival » Sat Dec 05, 2015 4:06 am

Okay, I read your 3 blog posts about terrain from November 2014 and I now understand both your replies a lot better.
So about what I want to acomplish... well, it's may sound a little crazy so bear with me:

I came across a very nice 10k resolution heightmap from Middle Earth (from the EM-DEM project). Me, being a great fan of Tolkien's work, got the map, adjusted a few things on World Machine, made a bunch of splat, normal and colormaps, and scaled the map properly according to the official maps from the books, wich gave me a insane terrain of about 3572.744 km x 3701.491 km.
Since I know a thing or two about making games in Unity I thought I might make a super simple game, just to walk in Middle Earth and watch the beautifull vistas from that world, as close as possible to the author's imagination that I could make with what I have:
Image
That was about four days ago ( :lol: ) and yesterday I came across DFTFU and now my brain is boiling with ideas for several other projects of mine, but this one in particular got me to downloading and tinkering with the DFTools. So I guess what I really want is to use that Middle Earth heightmap into Unity, a player controller, and some embelishment (like weather, day-night, trees...) just so people can walk around and watch something beautifull and sense the "real" scale of that world.

Now, I can already do all that by myself (albeit it wouldn't be as good), and I even tested if Unity could handle such imense world (divided into 25 terrains) with a floating origin script and the result was actually really good (I climbed one of the Misty Mountains, and there was no noticeable lag when the world re-centered).
But since you already created all these amazing features (like weather, the atlases system, calendar, player classes, the procedural placement of trees and stuff...) I decided to try and implement them onto the world I created.
That's not the only project in wich the DFTools would be usefull to me, but It's where I want to start. :oops:

So if its not possible to use your terrain system with my terrain, I would like to at least be able to use some of the other features*. So what you think?

* mainly those about climate, season, world time, billboard shader,texture atlas generation.

** Sry for wall of text

User avatar
Interkarma
Posts: 4863
Joined: Sun Mar 22, 2015 1:51 am

Re: Creating Custom Worlds

Post by Interkarma » Sat Dec 05, 2015 7:05 am

That sounds awesome! I would explore the hell out of a detailed Middle Earth, just so you know. :)

With the detail and sizes you have, it might be best just to lift some of the concepts from DFTFU, rather than use the library directly. As you've discovered, DFTFU is heavily bent towards a Daggerfall way of doing things and works best when creating content in that style. What you're doing is definitely a few notches above Daggerfall.

You could totally use the same concepts though, and you're welcome to take any of my code to use however you like in your project. Everything of mine is under MIT license, so you're free to use any of it for personal or commercial projects.

Maybe as a starting point, you could try feeding your height data into a DFTFU StreamingWorld using a custom TerrainSampler. You could use the same bicubic and noise techniques I use in DefaultTerrainSampler to smoothly resample that 10k heightmap to the appropriate dimensions. That will at least get you a world to walk around in without too much effort. From there, you could start to hack StreamingWorld to work how you need it. Then slowly decouple from Daggerfall and make it do things your way.

There's also a new Unity asset I've been playing with called Gaia. This isn't quite streaming in the way DFTFU does it, but it creates remarkably beautiful worlds from almost any data with very little effort. This video shows how it works. I imagine you could get your pre-generated world up and running in high detail very easily with that tool.

Good luck with everything! Please feel free to post updates here, whether you use DFTFU or not to accomplish things. I would be interested in watching your progress. :)

odival
Posts: 4
Joined: Fri Dec 04, 2015 9:27 pm

Re: Creating Custom Worlds

Post by odival » Sat Dec 05, 2015 5:52 pm

Thanks Interkarma

I take it that these are the lines in DefaultTerrainSampler that should be changed so I can feed my heightmap in right?

Code: Select all

int mx = mapPixel.mapPixelX;
            int my = mapPixel.mapPixelY;
            byte[,] shm = dfUnity.ContentReader.WoodsFileReader.GetHeightMapValuesRange(mx - 2, my - 2, 4);
            byte[,] lhm = dfUnity.ContentReader.WoodsFileReader.GetLargeHeightMapValuesRange(mx - 1, my, 3);
If so, how exactly can I change them to feed custom heightmap? I mean, I know I can just call a private function (sorry if the terminology is wrong, I'm still learning it) to load it from the Resources folder or just select it in the inspector, and set the resolution and heights... I'm just kinda lost as how to insert this kind of code inside your void GenerateSamples().

User avatar
Interkarma
Posts: 4863
Joined: Sun Mar 22, 2015 1:51 am

Re: Creating Custom Worlds

Post by Interkarma » Sun Dec 06, 2015 10:43 pm

Sorry for late reply. :)

Rather than modify DefaultTerrainSampler (which has a lot of Daggerfall-specific stuff), create a new sampler by copying everything in NoiseTerrainSampler (which has the basic loops setup) to a new class and register your sampler by setting the DaggerfallUnity.TerrainSampler property. This will give you basic rolling hills and demonstrate how the sampler works without any Daggerfall stuff getting in the way.

Once you have your own sampler running, you need to work out how to return your data in a meaningful way. You will need to slice your world up into 1000x500 "chunks" of HeightmapDimension*HeightmapDimension values. The GenerateSamples() member of ITerrainSampler interface is used to return the values for any particular chunk.

Since your source data is lower resolution than the overall world (which is minimum of 127k * 63.5k samples), you will probably need to stretch things out using bicubic filtering like I do with DefaultTerrainSampler. You could also just fit your source map into one corner and return 0-height terrain everywhere else.

In any case, have a play with the sampler concept before feeding in your own data so you can get a feel for how it works. For example, you could return an increasing scale for the X-Y axes per chunk. That should help demonstrate the concept and you can build on it from there.

Good luck! Let me know how you go.

Post Reply