Modding Tutorials: World Data Overrides

Discuss modding questions and implementation details.
User avatar
Hazelnut
Posts: 1695
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Modding Tutorials: World Data Overrides

Post by Hazelnut » Mon Oct 14, 2019 2:38 pm

Overriding Daggerfall World Data (WIP)

Introduction
The World Data Override system has been built so that mods can alter and add to the Daggerfall world data after it's streamed from the original data files, but before it's used by the DFU engine. This allows modifications of the gameworld without needing to change the original data files. The override files are json format and much easier to edit than the original binary files, and targetted sections of world data can be changed, while the remainder is unaltered. This also avoids the issue of these files being copyrighted by Bethseda, and the DFU model is to require the original files without modifications.

Comparison with the 'New Locations' mod
Uncanny Valley has created a new locations modding resource already, and it's been used to create new things in the Daggerfall gameworld, so why is overriding world data of any use you may ask? The location loader and other resources in 'new locations' are fantastic, but they serve a different purpose. It enables new content made in Unity to be layered on top of the gameworld, allowing new decorations and places to appear in the exterior overworld. It doesn't enable new interiors that are transitioned into to be added. I think the name locations is a bit misleading, since they're not daggerfall locations and are not integrated into the gameworld. Brilliant for adding small places of interest around the map, but not for a new town or dungeon.

Background info (must read and understand!)
The world of Daggerfall is specified in several bsa data files from the original game that DFU streams data from at runtime. These files define the overworld map, locations (towns & dungeons), buildings, dungeons etc that make up the gameworld. The world data override system allows selective parts of the location data to be overridden and changed.

There are over 15 thousand locations across the regions of the Iliac bay, constructed from reusable blocks. World blocks for the outside world (RMB - Record Map Block), which includes building interiors, and dungeon blocks for large interior spaces (RDB - Record Dungeon Block). Dungeons interiors are constructed from RDB blocks and towns & dungeon exteriors are constructed from RMB blocks. The locations are merged into the streaming world at the map pixel coordinates specified.

The hierarchy is illustrated in this diagram.
DFworldData.png
DFworldData.png (13.48 KiB) Viewed 42 times
Each map pixel is mapped to a region, and each region has a bunch of locations within it. Each location then has an overworld exterior constructed from reusable RMB blocks. As you can see from the illustration blocks can be used in multiple locations, so changing a block will potentially change many locations across the world and in different regions since all places the block is used will change. Locations can also optionally define a dungeon from RDB blocks, again the blocks can be shared between different locations.

Console data dump commands
To assist with overriding world data I've added several data dump console commands to DFU that dump various world data into JSON files. The files go into the folder where you can find the DFU settings file and do nothing at all if left there. They are simply put there for you to look at, copy from, move and edit etc. Just be aware that the contents will include any override data the game has loaded from mods and loose assets, these will need to be disabled to get a file with just the vanilla data.
  • dumpregion - Dump the current DFRegion to JSON file. (for example data only)
  • dumplocation - Dump the current DFLocation to JSON file. (can be used for location overrides)
  • dumpblock - Dump a set of RMB & RDB blocks to JSON files. e.g. "dumpblock RESIAM10.RMB M0000004.RDB" (can be used for RDB overrides) If no block names specified, will dump a block index to "BlockIndex.txt" containing all of the blocknames indexed as they are from the original data.
  • dumplocblocks - Dump the names of blocks for each location, or locations for a block, to JSON file. With no argument you get a list of blocks for all locations in the game which will take a while so be patient. Providing a blockname gives a list of all locations split by region that contain that block. e.g. "dumplocblocks RESIAM10.RMB" (useful for finding everywhere a block you're modifying is used) Giving the argument "locindex" dumps a location index to "LocationIndex.txt" with regionIndex, locationIndex, location name.
  • dumpbuilding - Dump the RmbSubRecord of current building player is inside to JSON file. e.g. "dumpbuilding"
Do I have to edit the raw json files?
Currently yes you do. Although Uncanny Valley has already made an building editor, and I believe has plans for a dungeon editor. Hopefully these will be released to the modding community and allow the json files for blocks to be editable using the Unity 3D editor. There will probably still be some json editing required, for locations for instance unless UV does an editor for that as well, but RMB and RDB edits where models and flats are being positioned will be a lot easier with a 3D editor.

What can I use to edit the raw json files?
Any text editor you like, but I would recommend one that can interpret json and allow section folding. This is incredibly helpful for navigating and focussing on specific parts of the data. My recommendation is to use VSCode which is free, provides json folding, and very good. Atom is another good option.

Development of override files.
When developing override files, the best way is to run the game in the Unity editor and put the files as loose files into StreamingAssets/WorldData/. This will allow changes to those files to be reflected in-game without requiring a restart. When not running in the editor, the override data will be loaded and cached for the rest of the session, so changes will not be seen in game until it's restarted.

Test files.
I've attached a zip file with the test files I've used when developing this feature. The new dungeon block references a custom model from the Archaeologists mod, so you will need that installed or you will need to change the model id to one from the original data otherwise the block layout will fail.
WorldDataTestFiles.zip
(7.05 KiB) Downloaded 4 times

Using Variants for a more dynamic Daggerfall world! (WIP)

Coming soon is a world data variant system which will allow different overrides to be used once a trigger happens. The triggers can set variant blocks to be used just for a specific location, or every location using that block like the base system does.

User avatar
Hazelnut
Posts: 1695
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Modding Tutorials: World Data Overrides

Post by Hazelnut » Mon Oct 14, 2019 2:39 pm

Locations

To be able to target a building within a location, the sector field of the building data records can be used to index the block name in the exterior data array. The correct block containing the building must be known so the building key can be calculated when referencing the building & location from the place definition in a quest script. This is not what the field was for, but as it's unused I re-used it to allow this.

Altering an existing location

To alter an existing location, travel there and run the dumplocation command to get the current data for that location. The json file will be called location-RR-NNNN.json where RR is the region index (e.g. 17 for daggerfall) and NNNN is the location index within that region. The json in this file can be edited to change the location in the same ways as shown for adding new locations. Region Ids can be found here: https://en.uesp.net/wiki/Daggerfall:Region_Numbers

At present, all that can be visibly changed for a location exterior is the RMB blocks used and their layout. This is done by changing the list of block names and the width / height elements. Only existing blocks can be used at this time until the 2D array serialization issue is resolved to allow entire RMB blocks to be changed or added. Individual buildings can have their interior and exteriors replaced, see later on in this tutorial for more detail. Building data is also at the location level and this is combined with the block building data at runtime to give the buildings (which are otherwise the same for every location the block is used) location specific data for nameSeed, factionId, and quality.


Altering a dungeon location dungeon block layout

Altering a dungeon location dungeon block layout is a case of specifying the RDB blocks in a list and defining the x,y coordinates they should be at. Only one block can be marked as the entrance block, and even though the crypt locations in original data all seem to define an '+' shape of 5 blocks with only the central one actually accessible by players, there's no need to do this for DFU and, as shown in my examples, just the single block seen by players needs to be defined. See https://en.uesp.net/wiki/Daggerfall:MAP ... ck_Element for more details.


Adding a new location

New locations should be named 'locationnew-LOCNAME-RR.json' where RR is the region index the new location will be in, and LOCNAME is your file name for this location. e.g. 'locationnew-archcrypt1-17.json' When adding a new location, I suggest finding a similar existing location and use the dumplocation command to get the data for it, then rename the file and edit it to become your new location as shown below. A unique location index will be assigned internally when loaded in the game, and this could be different each time depending on other mods, so you cannot rely on it being a particular value and should set the corresponding data elements to zero as shown below. You must restart the game when adding a new location replacement file regardless of running in the editor, as the index will not get assigned correctly. Once the new location file is there, you can make changes while the game is running as normal.

Now you will need to decide is where to put your new location on the map, first choosing a region, and then the specific map pixel in that region. As far as I know you can't have two locations in one map pixel. (utility to assist is coming... be patient: The easiest way to find the coordinates you want is to use the in-game travel map with the WorldDataUtilities.dfmod installed. This will display the map pixel coords the mouse is over enabling easy selection of a suitable map pixel.) Once you have the map pixel x,y coords then you can calculate the mapId and the other coordinates required for the header. These must be consistent otherwise errors will occur. I will refer to the map pixel x,y coords as mappx.x and mappx.y in all of the instructions below and I will use the example numbers 123,456 for mappx.x,y below.

Code: Select all

1) MapId = (mappx.y * 1000) + mappx.x = 456123
2) ExteriorData.MapId = MapId 
    (In original data, the MapId & 0xfffff = ExteriorData.MapId with the upper bits contain the LocationIndex but 
     these upper bits are unused by DFU so you can simply set both MapId's the same)
3) Latitude = (499 - mappx.y) * 128 = 5504
4) Longitude = mappx.x * 128 = 15744

Annotated example: a new crypt location

This is an annotated example of a new crypt location from the test files zip. The filename is: "locationnew-archcrypt1-17.json" and the 17 is the region index of Daggerfall. It's a copy of the Gaerhart Vaults and is placed one map pixel to the east, and uses a new added block ARCHC001.RDB for the crypt interior. It also uses a different existing RMB block for the exterior.

Location-crypt.jpg
Location-crypt.jpg (350.78 KiB) Viewed 740 times

Annotated example: existing shack location

This is an annotated example of an existing shack location from dumplocation command. The filename is: "location-17-1268.json" and the 17 is the region index of Daggerfall, and the 1268 is the location index. It shows a location with building data, and no dungeon.

Location-shack.jpg
Location-shack.jpg (186.47 KiB) Viewed 821 times

User avatar
Hazelnut
Posts: 1695
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Modding Tutorials: World Data Overrides

Post by Hazelnut » Mon Oct 14, 2019 2:41 pm

Dungeons

Altering an existing dungeon block

To alter an existing dungeon block, first dump the original using the dumpblock command. Do ensure you don't accidentally get the data from installed mods, unless that's what you want to start with. The file will be called BLOCKNAME.RDB.json and can be edited then moved into WorldData. See the annotated examples for details of how to change the data, but in summary there's a list of model references and a list of lists of rdbObject's. These objects can be flats, lights or models that make up the block. It's unclear why this is a list of lists, and I've not spotted any pattern yet. I suggest that any new objects are added to one of the empty lists to make it easier to keep track of new stuff, but it doesn't really matter. The model refs are used by model objects by the model ref list index, so any models you want to use need to have a reference with a numeric model id. New models provided via the asset injection system can have references added, but only if the model has a numeric name. (try to avoid collisions between mods)

In addition, model objects can have action resources attached to enable the model to trigger an action when clicked by the player. I've not figured out how this all works, but these can be added. It's unnecessary for any models that don't have actions, so feel free to delete it.


Adding a new dungeon block

To create a new dungeon block you will need a name for it, e.g. ARCHC001.RDB from my examples. I don't think it has to follow the 8.3 format of classic block names, but I've not tested that. Feel free to try and report back if you want. Next either start with an existing block dump and delete everything you don't need. The dungeon block will need constructing from existing and new models, and each will need a single entry in the model ref list regeardless of how many times the model is used. Then create object list entries for each placement of the model, following up with entries for flats and lights. Currently there's no taxonomy of the existing models in Daggerfall so it can be hard to find what you are looking for, but they can be browsed using the modelling utility availiable here.

Note that you cannot use a new model for the entrance part of the block without creating a related instance of CustomDoor.cs, otherwise it has to be an existing model with a static exit door definition. Also worth mentioning that a good way to look at complete blocks is to run DFU in the Unity editor and navigate to a dungeon the block is used in, pause the game, then switch to the scene view. Click on the interior dungeon tree on the left panel and select the block entry you're interested in. Rendered blocks have all the models combined so the entire 'finished' block can be inspected and rotated etc.


Annotated example: a new crypt block

This is an annotated example of a new crypt block from the test files zip. The filename is: "ARCHC001.RDB.json" and is referenced from the new location data as shown above.

RDBblock.jpg
RDBblock.jpg (195.28 KiB) Viewed 736 times
RDBblockObjectLists.jpg
RDBblockObjectLists.jpg (74.37 KiB) Viewed 736 times
RDBblockObjects.jpg
RDBblockObjects.jpg (298.93 KiB) Viewed 734 times

User avatar
Hazelnut
Posts: 1695
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Modding Tutorials: World Data Overrides

Post by Hazelnut » Mon Oct 14, 2019 2:46 pm

Towns

The RMB blocks used for overworld locations like towns cannot currently be overridden, only individual buildings in an RMB block can be altered. This is because the serializer doesn't handle the 2D array used for the ground data. I'm sure it can be extended to do so if there's enough demand for this feature.

UESP has a useful set of pages showing the existing blocks of the game here: https://en.uesp.net/wiki/Daggerfall:Blo ... apPItem.29

Altering an existing town building

Note: Must set faction id if this is a new 'named' building (see RMBLayout.IsNamedBuilding()) but leave it zero when not named building or changing an existing one.

BadLuckBurt
Posts: 98
Joined: Sun Nov 05, 2017 8:30 pm

Re: Modding Tutorials: World Data Overrides

Post by BadLuckBurt » Wed Oct 16, 2019 8:15 pm

I could follow most of the annotated example for a dungeon. I just have a few questions :oops:

Exterior > RecordElement > Header
Unknown2 is used to overwrite an existing location when you copy from LocationIndex right?
How do we know what the current max number is for LocationId and how far up is the number allowed to go?

Dungeon > Blocks

Code: Select all

Ex: Privateer's Hold
           B0000012          
 B0000009 @S0000999  B0000003
           B0000006  
The ARCHCH001.RDB file that's in your example would just be one of the B0000012, 03, 06, 09 or @S0000999 (when set as starting block in the above setup right, the position depending on their X-Y coordinate?

I don't understand what the '5 blocks defined but only the center one is used so they can be cut down to just that block' means exactly. Does the ARCHCH001.RDB offer different models but only 1 is usable?

User avatar
Hazelnut
Posts: 1695
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Modding Tutorials: World Data Overrides

Post by Hazelnut » Wed Oct 16, 2019 8:32 pm

BadLuckBurt wrote:
Wed Oct 16, 2019 8:15 pm
I could follow most of the annotated example for a dungeon. I just have a few questions :oops:

Exterior > RecordElement > Header
Unknown2 is used to overwrite an existing location when you copy from LocationIndex right?
How do we know what the current max number is for LocationId and how far up is the number allowed to go?
Unknown2 should simply have the value that LocationIndex at the end of the file has. This will be defined and fixed if you're changing an existing location and should be set to 0 for new locations since the value will be auto-assigned. Guess my explanation text was a bit cryptic.

LocationId is a 16 bit unsigned int, but you could find that out from the DFLocation class if you needed to know.

Regarding the single dungeon block, I was just pointing out that a dungeon with 1 block only needs that block in the list, whereas DF data has 4 additional blocks for these cases as you will see when dumping the data for a crypt.

BadLuckBurt
Posts: 98
Joined: Sun Nov 05, 2017 8:30 pm

Re: Modding Tutorials: World Data Overrides

Post by BadLuckBurt » Wed Oct 16, 2019 9:02 pm

Hazelnut wrote:
Wed Oct 16, 2019 8:32 pm
Unknown2 should simply have the value that LocationIndex at the end of the file has. This will be defined and fixed if you're changing an existing location and should be set to 0 for new locations since the value will be auto-assigned. Guess my explanation text was a bit cryptic.

LocationId is a 16 bit unsigned int, but you could find that out from the DFLocation class if you needed to know.

Regarding the single dungeon block, I was just pointing out that a dungeon with 1 block only needs that block in the list, whereas DF data has 4 additional blocks for these cases as you will see when dumping the data for a crypt.
Thank you, I should've guessed 0 would make the LocationIndex auto-assigned, makes sense.

I know the LocationId is only used when you specifically target a location in a quest but I was just wondering what would happen when two people pick the same location id for different locations?

Sorry for bugging you with these questions, I'm going to give setting up a new dungeon a shot this Saturday and I like to understand the 'why' of things I'm doing, helps to troubleshoot too.

User avatar
Yagiza
Posts: 99
Joined: Wed Jul 31, 2019 5:16 pm

Re: Modding Tutorials: World Data Overrides

Post by Yagiza » Sat Nov 09, 2019 3:19 pm

Hazelnut wrote:
Mon Oct 14, 2019 2:46 pm
Towns

The RMB blocks used for overworld locations like towns cannot currently be overridden, only individual buildings in an RMB block can be altered. This is because the serializer doesn't handle the 2D array used for the ground data. I'm sure it can be extended to do so if there's enough demand for this feature.

UESP has a useful set of pages showing the existing blocks of the game here: https://en.uesp.net/wiki/Daggerfall:Blo ... apPItem.29

Altering an existing town building
Is it possible to add a new town on the world map? I tried and failed.
I just dumped Ripwych Commons of Daggerfall and modified the file, following the manual above. When I try to use the file I got, the game just hangs on startup. When I run it in Unity Editor, I see that System.NullReferenceException trown in ExteriorAutomap.OnMapPixelChanged(), 'cause ContentReader is null.

I attached the file I try to use.
Attachments
locationnew-test-17.zip
(1022 Bytes) Downloaded 3 times
Last edited by Yagiza on Sat Nov 09, 2019 6:52 pm, edited 1 time in total.

BadLuckBurt
Posts: 98
Joined: Sun Nov 05, 2017 8:30 pm

Re: Modding Tutorials: World Data Overrides

Post by BadLuckBurt » Sat Nov 09, 2019 3:42 pm

Yagiza wrote:
Sat Nov 09, 2019 3:19 pm
Is it possible to add a new town on the world map? I tried and failed.
I just dumped Ripwych Commons of Daggerfall and modified the file, following the manual above. When I try to use the file I got, the game just hangs on startup. When I run it in Unity Editor, I see that System.NullReferenceException trown in ExteriorAutomap.OnMapPixelChanged(), 'cause ContentReader is null.

I attached the file I try to use.
It's possible but it doesn't work as expected, the new town will be a clone of the old town. Changing the NameSeeds will give you different building names but the interiors link back to the original town.

I don't know about running it from the editor, I just have a build of Hazelnut's branch and that seems to work fine. I tried loading in your location and didn't get any errors but since I don't know where it is located, I couldn't go there. Tried search for 'Test' but that sent me to Coppersfield Manor, not your location.

I've put one of my own experiments on Pastebin: https://pastebin.com/11sKjtqN. You can try loading that one, just name it locationnew-burtbury-17.json or something. It's a copy of Midbury in Daggerfall.

The only way to get a unique location going is if you make your custom RMB blocks which is in the works but not documented at all.

User avatar
Hazelnut
Posts: 1695
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Modding Tutorials: World Data Overrides

Post by Hazelnut » Sat Nov 09, 2019 3:50 pm

I will need more info about what you did to be able to help. Where did you place the file? When did the error occur? What version of the DFU codebase are you using, the worldDataDev test branch?

Post Reply