Links to Sections Introduction
The World Data Override system has been built so that mods can alter and add to the Daggerfall world data after it's streamed from the original data files, but before it's used by the DFU engine. This allows modifications of the gameworld without needing to change the original data files. The override files are json format and much easier to edit than the original binary files, and targetted sections of world data can be changed, while the remainder is unaltered. This also avoids the issue of these files being copyrighted by Bethseda, and the DFU model is to require the original files without modifications.
I know this tutorial is more of an info dump and is not easy to read & comprehend, but please do read it. I've tried to write down what I know in the best way I can and have spent hours doing this when I would much rather have been coding and solving problems. Making overrides will require effort, time and understanding. The more you're pushing the boundaries of what's already been done before, the more you will need to understand the data structures and the code that uses them - so you are aware of the various constraints and can problem solve to reach a solution. I'll help as much as i can, but I don't know everything and my answers will often involve me reading through code and figuring stuff out so you may need to be patient. (Hint: queries with all relevant detail and context provided, and demonstrating the effort taken to understand and solve yourself will always get a response )
What is World Data?
By world data I'm referring to the data that defines the locations of the Daggerfall game world. All the coloured pixels that you see on the games travel map and select as destinations to fast travel to. Basically towns, villages, dungeons etc. These locations are merged into the world terrain, which is a completely different system where a TerrainSampler uses the classic heightmap data to generate the terrain of the world. This terrain is not something that can be altered by the world data system discussed here. DFU basically flattens the terrain to place a locations' exterior and paints over the texture tiles of the terrain with texture tiles defined in the world data.
Much of the DFU code relies on the world data being as expected. There are constraints on what you can change without breaking DFU, so care needs to be taken to change the world data only in ways that DFU expects. Following existing patterns in the classic data is the safest way to ensure this. For newcomers I would recommend starting with building overrides as they're the easiest to work with and the most useful. (see the last section of this guide "Altering an existing town building")
Background info (must read and understand!)
The world of Daggerfall is specified in several bsa data files from the original game that DFU streams data from at runtime. These files define the overworld map, locations (towns & dungeons), buildings, dungeons etc that make up the gameworld. The world data override system allows selective parts of the location data to be overridden and changed.
There are over 15 thousand locations across the regions of the Iliac bay, constructed from reusable blocks. World blocks for the outside world (RMB - Record Map Block), which includes building interiors, and dungeon blocks for large interior spaces (RDB - Record Dungeon Block). Dungeons interiors are constructed from RDB blocks and towns & dungeon exteriors are constructed from RMB blocks. The locations are merged into the streaming world at the map pixel coordinates specified.
The hierarchy is illustrated in this diagram.
Each map pixel is mapped to a region, and each region has a bunch of locations within it. Each location then has an overworld exterior constructed from reusable RMB blocks. As you can see from the illustration blocks can be used in multiple locations, so changing a block will potentially change many locations across the world and in different regions since all places the block is used will change. Locations can also optionally define a dungeon from RDB blocks, again the blocks can be shared between different locations.
Console data dump commands
To assist with overriding world data I've added several data dump console commands to DFU that dump various world data into JSON files. The files go into the folder where you can find the DFU settings file and do nothing at all if left there. They are simply put there for you to look at, copy from, move and edit etc. Just be aware that the contents will include any override data the game has loaded from mods and loose assets, these will need to be disabled to get a file with just the vanilla data.
- dumpregion - Dump the current DFRegion to JSON file. (for example data only)
- dumplocation - Dump the current DFLocation to JSON file. (can be used for location overrides)
- dumpblock - Dump a set of RMB & RDB blocks to JSON files. e.g. "dumpblock RESIAM10.RMB M0000004.RDB" (can be used for RMB & RDB overrides) If no block names specified, will dump a block index to "BlockIndex.txt" containing all of the blocknames indexed as they are from the original data.
- dumplocblocks - Dump the names of blocks for each location, or locations for a block, to JSON file. With no argument you get a list of blocks for all locations in the game which will take a while so be patient. Providing a blockname gives a list of all locations split by region that contain that block. e.g. "dumplocblocks RESIAM10.RMB" (useful for finding everywhere a block you're modifying is used) Giving the argument "locindex" dumps a location index to "LocationIndex.txt" with regionIndex, locationIndex, location name.
- dumpbuilding - Dump the RmbSubRecord of current building player is inside to JSON file. e.g. "dumpbuilding"
Uncanny Valley has created a new locations modding resource already, and it's been used to create new things in the Daggerfall gameworld, so why is overriding world data of any use you may ask? The location loader and other resources in 'new locations' are fantastic, but they serve a different purpose. It enables new content made in Unity to be layered on top of the gameworld, allowing new decorations and places to appear in the exterior overworld. It doesn't enable new locations that can be fast travelled to or used in quest scripts however. I think the name locations is a bit misleading, since they're not daggerfall locations and are not integrated into the gameworld. Brilliant for adding small places of interest around the map, but not for a new town or dungeon.
Do I have to edit the raw json files?
Location and RMB blocks require json file editing. Buildings (RMB subrecords) and Dungeons (RDB blocks) can be edited using the 3D WorldData Editor (soon to be) supplied with the Daggerfall Unity source. Buildings edited can be put together into a complete RMB override if needed, however this is done via json editing.
What can I use to edit the raw json files?
Any text editor you like, but I would recommend one that can interpret json and allow section folding. This is incredibly helpful for navigating and focussing on specific parts of the data. My recommendation is to use VSCode which is free, provides json folding, and very good. Atom is another good option.
Development of override files.
When developing override files, the best way is to run the game in the Unity editor and put the files as loose files into StreamingAssets/WorldData/. This will allow changes to those files to be reflected in-game without requiring a restart, just re-load or exit and re-enter to get DFU to re-build the scene. When not running in the editor, the override data will be loaded and cached for the rest of the session, so changes will not be seen in game until it's restarted.
Test files.
I've attached a zip file with the test files I've used when developing this feature. The new dungeon block references a custom model from the Archaeologists mod, so you will need that installed or you will need to change the model id to one from the original data otherwise the block layout will fail. You can also see all the source files for my mods in github here: https://github.com/ajrb/dfunity-mods