CommunityUs Release

Off topic discussion. Talk about gaming and life in general. Be awesome to each other.
communityus
Posts: 41
Joined: Fri Apr 05, 2019 1:51 am

CommunityUs Release

Post by communityus » Sat Apr 13, 2019 9:01 am

I felt it was time to reformat this post. So what is this then??
Sometime in 2021: I lowercase and more humble communityus will endeavor to release a new game using DFU. DFU will be used as a game engine/framework and no Intellectual Property will be borrowed or derived from proper DF. I have 2 years, so should be just enough time to sort that all out with the help of others of course!
Below will be milestone posts and likely some unrelated content documenting the journey.

PICTURES coming soon! Like in 2 months soon.


[Original Post: Sat Apr 13, 2019 3:01 am]
Hi everyone,

A few of you may have seen my Legal Question post here:
viewtopic.php?f=4&t=181

After hashing out all the nitty gritty details above - myself and 1x artist have decided to make a go at a branch of this engine for production purposes.

The majority of the work has already been done by everyone here esp. Interkarma to approximate interaction with arena2 file types - what we don't have is our own copyleft or similar non Bethesda IP (Intellectual Property) file set for arena2, with close specifications so a fully new product can be made without strings attached - aside from MIT libre use for not only the engine but default file set that avoid derivative pitfalls by fitting existing open art into the provided template and created by an artist with no ties to this project. (Retrofitting)

What we have done thus far:
Stripped required file set down from ~1700 items (the default in arena2) to only 30 files to render a chunk with houses, trees, animated cows and a few signs.

Note this is the default model in the DaggerfallUnity Component Script, however it now loads a different block set. After further refinement and content replacement we will document the spec and standardize it free of any IP.

ModelID:
456

Block Name:
CUSTAA06.RMB (remapped to our cow scene)

Let us call it, "Isle of Ball/Dice Twice" for now.

We also have reduced the file types needed to only 7.
  • 20x TEXTURE.???'s
  • 3x BSA's (ARCH3D/BLOCKS/MAPS - blocks and maps have been reduced to less than 5MB collectively, nothing done yet to reduce ARCH3D)
  • 2x UI's for the mod menu (ICON00I0, MASK04I0)
  • 1x ART_PAL.COL
  • 1x SKY??.DAT
  • 1x DAGGER.SND
  • 1x WOODS.WLD (only 40KB!)
  • 1x FLATS.CFG
This means we've pulled about 75MB out of BLOCKS/MAPS/WOOD files.
As a bonus we have verified CLIMATE.PAK and POLITICS.PAK work, they have been shrunk to 1KB only! Just haven't lined it up with WOODS.WLD yet so everything is from same area and demoing a large feature set of the engine from our own files.

Below is a screenshot, of course what is left is still Bethesda IP but this is all we have left to strip out and replace with open source IP or open licensed stuff from the community here.

The first goal will be to allow anyone to launch a pixel art level project, then fast coming support for all the mod types here esp. the open source ones (all thus far?)

We are pushing hard to wrap up a demo over the next few weeks. I may ask a few of you for use of your screenshots for promotional pieces. This post will serve as the place to discuss matters related to this Community Project. We don't plan to highjack the forums here, as our use case differs but in the overlap we will chat about anything related.
Attachments
progress.PNG
progress.PNG (788.07 KiB) Viewed 984 times
Last edited by communityus on Tue Feb 11, 2020 5:45 am, edited 5 times in total.

BadLuckBurt
Posts: 431
Joined: Sun Nov 05, 2017 8:30 pm

Re: CommunityUs Release

Post by BadLuckBurt » Mon Feb 10, 2020 9:27 am

communityus wrote:
Mon Feb 10, 2020 9:11 am
Thx for clearing that up - so no blender exporter for RMB / RDB blocks - but to the JSON spec on this thread. Might be just as well. In early stages I plan to use location loader anyway until I can figure out the deep dive stuff.

I agree - my own data structures will evolve naturally with the project. I'm not against them, just not top of the list when we have all these tools! ;-)
I decided to reply here, I know Hazelnut would like the WorldData thread to remain informative instead of a discussion thread.

Do you mean Uncanny Valley's location loader? Because that one works a bit different from WorldData.

communityus
Posts: 41
Joined: Fri Apr 05, 2019 1:51 am

Re: CommunityUs Release

Post by communityus » Mon Feb 10, 2020 9:38 am

Yes the Uncanny one. I also was looking at some old mesh replacement work I can link too. All different approaches for different problems. I am following them all to be sure. Since right now I am mainly just world building and getting my feet wet. Location Loader is good stop gap until I get the other stuff sorted out in my brain. :-)

BadLuckBurt
Posts: 431
Joined: Sun Nov 05, 2017 8:30 pm

Re: CommunityUs Release

Post by BadLuckBurt » Mon Feb 10, 2020 9:45 am

Cool, as long as you're aware of the differences :)

Uncanny's location loader is great for adding stuff to Daggerfall's world that's not a location. Kamer has used it for Rocks of Daggerfall to great effect to make the wilderness more interesting.

To eventually edit existing and add new locations (selectable on the travelmap, in quests, etc.), WorldData is the way to go.

communityus
Posts: 41
Joined: Fri Apr 05, 2019 1:51 am

Re: CommunityUs Release

Post by communityus » Tue Feb 11, 2020 5:38 am

Uncanny_Valley wrote:
Fri Dec 20, 2019 12:10 am
Hazelnut wrote:
Thu Dec 19, 2019 11:06 pm
I think you should make it clear that it can only edit building world data files (blockName.RMB-nn-buildingX.json) and not locations or RMB / RDB blocks. Or am I doing something wrong - I got exceptions for anything that wasn't a building.
It only supports editing buildings (or interiors as I refer to them), I tried to make it clear in the first line of the post, but I can see if I can make it clearer.
Hazelnut wrote:
Thu Dec 19, 2019 11:06 pm
Also loading buildings which use new models causes an exception because the model is not part of the DF data. No idea how this could be solved. Personally I do the editing first and then switched the model later, but that's cos I only have 1.
The editor has been a work in progress during my work with the taverns, so adding support for custom new models hasn't been relevant thus far for me. One solution could be that if the editor finds and object that isn't part of the orginal DF data, it still loads the objects but uses some "unknown object" token as its visual representation in the editor. So you can still edit and work with it, even if you don't get a full visual representation. I would like to be able to load up the editor with mods, to be able to preview the location with texture and model replacements, but haven't gotten that far it. As mentioned, its far from finished.
Hazelnut wrote:
Thu Dec 19, 2019 11:06 pm
Could we get a ground plane for the building exteriors? It's really hard to place objects without being able to see where the ground is.
That's a really good idea, and should be easily fixed. I just wonder how big it should be? Do building have a fixed size or does it vary?
Hazelnut wrote:
Thu Dec 19, 2019 11:06 pm
Lastly, how can I switch it off when i want to just run DFU? Just discard changes for the DF startup scene?
You can either select the parent object in the Hierarchy and just delete it, or closing down the editor (as in right click and "close tab" on the editor window).
BadLuckBurt wrote:
Mon Feb 10, 2020 9:45 am
Cool, as long as you're aware of the differences :)

Uncanny's location loader is great for adding stuff to Daggerfall's world that's not a location. Kamer has used it for Rocks of Daggerfall to great effect to make the wilderness more interesting.

To eventually edit existing and add new locations (selectable on the travelmap, in quests, etc.), WorldData is the way to go.
I am seeing the differences now. Thanks for pointing that out.

communityus
Posts: 41
Joined: Fri Apr 05, 2019 1:51 am

Re: CommunityUs Release

Post by communityus » Tue Feb 11, 2020 5:44 am

BadLuckBurt wrote:
Mon Feb 10, 2020 9:27 am
communityus wrote:
Mon Feb 10, 2020 9:11 am
Thx for clearing that up - so no blender exporter for RMB / RDB blocks - but to the JSON spec on this thread. Might be just as well. In early stages I plan to use location loader anyway until I can figure out the deep dive stuff.

I agree - my own data structures will evolve naturally with the project. I'm not against them, just not top of the list when we have all these tools! ;-)
I decided to reply here, I know Hazelnut would like the WorldData thread to remain informative instead of a discussion thread.

Do you mean Uncanny Valley's location loader? Because that one works a bit different from WorldData.
Agreed, I'll just move it over.
BadLuckBurt wrote:
Mon Feb 10, 2020 5:56 am
communityus wrote:
Mon Feb 10, 2020 5:33 am
This is perfect. I started creating my own visual version of this. But there is a ton in here I didn't even know. Huge time save! Thanks. I'd be happy to contribute to and help establish a wiki of sorts on GitHub or UESP somewhere.

To...um...add?...to the convo. Here is a horrible visual aid (shows some of how I have mapped the engine/framework workflow) - its super polished! :P


Spoiler!
pic 1: -> How MAPS.BSA plays a role. This may not be right but the next one about blockData was getting closer to what I think is going on data flow wise. I will leave this pic here for now so someone can correct me. If anyone can follow it that is.
Image

pic 2: -> blockData -> I think I nailed this.
Image

pic 3: -> As one plays around with removing all Daggerfall IP from the arena2 files and replacing the content, things break. Here is one..."what happens if". Note the ARCH3D.BSA file size at 463 kb and maps.bsa smaller too.Image

Playing in a iHex for now. With the documention provided above I am beginning to wonder if I can just replace the model data in the RDB/RMB files (etc) with random values and then using the override feature above replace all IP faster with this JSON approach.

Lots to think on. Thanks again for taking the time. I will use it for sure!
Your images are missing? And what do you mean by IP?
pic 1: -> How MAPS.BSA plays a role. This may not be right but the next one about blockData was getting closer to what I think is going on data flow wise. I will leave this pic here for now so someone can correct me. If anyone can follow it that is.
Image

pic 2: -> blockData -> I think I nailed this.
Image

pic 3 (similar to pic 1) and 4 (begin to see how one might sort this all out): -> As one plays around with removing all Daggerfall IP from the arena2 files and replacing the content, things break. Here is one..."what happens if". Note the ARCH3D.BSA file size at 463 kb and maps.bsa smaller too. From this we can infer, our BLOCKS.BSA (now 4 MB and stripped down) does not have the right RDB but we can fix that or using this document (above) perhaps fill the slot with random values and override it with JSON to have the work be less tedious when developing.Image
Image
communityus wrote:
Mon Feb 10, 2020 6:02 am
Had to clean the formatting up. Thank you for the reply though, all fixed now. (still learning the forum format so moved the :attachments: off and put them on GitHub where I can further use them.)

IP = Intellectual Property. Only a concern for those of us that are using it as a framework/game engine - not something normal modders/workshop folks would worry over.
BadLuckBurt wrote:
Mon Feb 10, 2020 6:15 am
Ah I see.

You might want to check the UESP pages on the files and data structures, think they're in the Hacking section. That is if you havent already ofcourse.

Our of curiosity, what are your goals?
communityus wrote:
Mon Feb 10, 2020 6:31 am
"...what are your goals?" - BadLuckBurt (There are some buggy things happening on MS EDGE so just doing quote like that.)

Beyond total conversion mod - but something on that level, to this extent:
using the codebase alone and none of DF original content (so that all needs to be replaced.)

Basically coming back after a while to try my goal again:
communityus wrote:
Sat Apr 13, 2019 9:01 am
...myself and 1x artist have decided to make a go at a branch of this engine for production purposes.
feel free to chat there, I don't want to derail this thread or anything.

Basically use DFUnity as a game engine / framework - because it has a ton of features and is well developed and well engineered and open source MIT (etc. etc. good things here) very little tech debt from the world go and so on. Just can't reuse anything from the original game. And that gets hard, but when you dig this deep (i.e. world data overrides) you can do it I've found. All the tools are out there to run your own pipeline without any DF IP content. It may take someone a week just orginzing their workspace to begin - but I haven't found a single file we can't replace with our own content. (note: it also can't be derived in someway from DF.)
BadLuckBurt wrote:
Mon Feb 10, 2020 7:03 am
Thanks for the link, I dont remember ever reading that post.

I knew it was theoretically possible to do what you are doing, restructuring the data files to basically create a new world but thats quite the undertaking.

It might be easier in some cases to develop your own data structure and readers for some things rather than stick to Daggerfalls formats.

Models for example can be replaced with modern meshes that work in Unity, the original meshes get converted as DFU reads them from the BSA since the way they are stored means they are not game ready for Unity.

I have written a couple of tools in HTML and Javascript to aid with WorldData manipulation, they're not completely done yet but you can find them here: https://github.com/BadLuckBurt

They're all very much geared towards Daggerfall itself atm (mainly regions, dungeon / location types and the available RMB / RDB blocks) but adapting them to your needs shouldnt be too hard.

They take away the need to edit certain things by hand.

Ive also created some automapping definitions for the terrain textures to use in Tiled and an export script to get them back into the RMB format, this allows you to edit the terrain textures in RMB blocks.

Lastly, I have some scripts for Blender that let you import RMB and RDB json files into your scene. I also have a prototype for exporting which I will finish when I make a custom dungeon.

If you are interested in any of those, let me know
communityus wrote:
Mon Feb 10, 2020 8:16 am
Very, very interested in the exporter. One of my main troubles is I can build the BSA's again, with MakeBSA but I have no way to replace the models (RMB / RDB blocks) in there yet. Below is what I am doing presently, starting with small BSA's manually adding the RMB/RDB's needed and making notes and thinking about my own art.

zoom in
Image

Hence my interest in World Data Overrides. So I was thinking I could remove the IP with random values as a hack for now, but would much prefer a blender tool that exported to the DF data structure that we could then put into the BSA's the proper way!



I get what you are saying about creating my own data structure. Two fold...[fold 1] there tends to be a lot of hidden overlap as one journeys down the rabbit hole with the DF data. Woods.WLD talks to Politic and Climate paks. And also talks to BLOCKS or ARCH3d or both - or both are required to sync in lockstep or the runtime halts because it can't find something in the other even if maybe MAPS (another one) didn't need that yet, something referenced it...etc. So you have all these files streaming in concert - hate to break up the band :-) and I didn't even get to quests yet and images also all crosstalking - don't forget save files too and likely other stuff PAL palates and skys get reference in other places and so on. So to create my own data structure is a thought, but a lot of work while we have existing tools and source code for all of them I think to really build out content creation tools down the road. So that is more on the table than creating my own data structure. Rather have things like your blender tool, that work to BSA way of doing stuff.

[fold 2]
I like there is a standard here. The minutiae is fun! It is a way of world building long lost I think. And with these tools its like it is setup to build while you are playing. I love that! This workflow is tested and refined, birthed from a production workflow.

I also like that it is an open source standard (DFWorkshop MIT and the XNA tools) and was designed to work on minimal spec hardware. There are a lot of things that can be done working from such a spec. once mastered. I feel it was engineered the old way.

I've found there is hidden value in doing things the old ways sometimes. I come from a video background - we used to have to wait for the edit to render. This gave us time to … wait for it … think. So our brains were already planning our next edits in our heads. And that is where the art comes from - our heads and thinking not clicking and instant seeing. VFX vs. creative story telling. (Both I say!)

I think this path will do much the same thing. The design forces one to think one way, with constraints. Also it sort of has a open source AssetBundle worked in - if you use the .BSA files. And yet best of both worlds, can use AssetBundles too for the dfmods, or like I say one day roll back the code if need to pivot away from Unity and still have a solid code base to start from. And this lets us be modular and not tied to Unity in the long wrong. As things further develop, there is already a lot done to work on a framework like XNA - which has a C++ similar library and so.
BadLuckBurt wrote:
Mon Feb 10, 2020 8:42 am
Sorry, I shouldve been more clear when I spoke of exporting, I meant export to DFU JSON format. While the models in BSA format are perfectly readavle thanks to Interkarmas work, converting modern meshes to a BSA format is far beyond my capabilities and probably more trouble than its worth. Especially UV data is stored in an odd way.

You do have a point about being able to drop DFU and do it in a more standalone way but since the mesh replacement uses FBX format, it should be trivial to adapt any solution you decide to use to work with that.

And true, the data is very interdependant which in turns makes it less flexible too. Creating yoyr own data structure is a bit of work but I think the flexibility gained from it would save time in the end. But yoy can always decide to take a look at that should you run into a wall, atm you seem to be making good progress regardless
communityus wrote:
Mon Feb 10, 2020 9:11 am
Thx for clearing that up - so no blender exporter for RMB / RDB blocks - but to the JSON spec on this thread. Might be just as well. In early stages I plan to use location loader anyway until I can figure out the deep dive stuff.

I agree - my own data structures will evolve naturally with the project. I'm not against them, just not top of the list when we have all these tools! ;-)
BadLuckBurt wrote:
Mon Feb 10, 2020 9:33 am
communityus wrote:
Mon Feb 10, 2020 9:11 am
Thx for clearing that up - so no blender exporter for RMB / RDB blocks - but to the JSON spec on this thread. Might be just as well. In early stages I plan to use location loader anyway until I can figure out the deep dive stuff.

I agree - my own data structures will evolve naturally with the project. I'm not against them, just not top of the list when we have all these tools! ;-)
I replied in your own thread: viewtopic.php?f=8&t=1986&p=39739#p39739 as the discussion is more about that than the actual WorldData system atm :)

If a mod could merge our last few posts with that thread, that'd be great, my thanks in advance.
Last edited by communityus on Tue Feb 11, 2020 10:23 pm, edited 4 times in total.

BadLuckBurt
Posts: 431
Joined: Sun Nov 05, 2017 8:30 pm

Re: CommunityUs Release

Post by BadLuckBurt » Tue Feb 11, 2020 6:04 am

Uncanny Valley actually has two seperate location thingies going. The one that you quoted from is the interior editor that he used to do the Taverns mod and is meant for editing building interiors and uses the WorldData system to override said building configurations.

There is also another one that deals with the overworld.

The overworld one doesnt use WorldData at all and thats the one I thought you were referring to. You can find it here: viewtopic.php?f=27&t=1046

communityus
Posts: 41
Joined: Fri Apr 05, 2019 1:51 am

Re: CommunityUs Release

Post by communityus » Tue Feb 11, 2020 9:28 pm

BadLuckBurt wrote:
Tue Feb 11, 2020 6:04 am
The overworld one doesnt use WorldData at all and thats the one I thought you were referring to. You can find it here: viewtopic.php?f=27&t=1046
lol, yup I think I mixed the names up.

also you were right to want to move our stuff over...I went ahead and changed all my text to "moved" might want to do the same and then apologized to Hazel.

BadLuckBurt
Posts: 431
Joined: Sun Nov 05, 2017 8:30 pm

Re: CommunityUs Release

Post by BadLuckBurt » Tue Feb 11, 2020 9:38 pm

communityus wrote:
Tue Feb 11, 2020 9:28 pm
lol, yup I think I mixed the names up.

also you were right to want to move our stuff over...I went ahead and changed all my text to "moved" might want to do the same and then apologized to Hazel.
Yeah, I cleared mine as well, would prefer to delete them but can't for some reason. Probably a time limit for that.

As long as you're aware there's two separate ways of adding stuff to the world, all good :)

communityus
Posts: 41
Joined: Fri Apr 05, 2019 1:51 am

Re: CommunityUs Release

Post by communityus » Thu Feb 13, 2020 4:57 am

Hi again. Thanks again for your help. I believe I have a clear understanding now of the differences.

I stumbled on this post:
BadLuckBurt wrote:
Thu Dec 19, 2019 8:57 pm
Ferital wrote:
Thu Dec 19, 2019 8:34 pm
...

If it helps with UV fixes, I have a library of all Daggerfall's models in .blend and .dae format as exported from Daggerfall Modeling although it sounds like you are manipulating the coordinates directly to get the desired result shown in DM?
Combined with your mention of blender tools for .rmb and .rdb I was wondering if you still have the blender models all dumped out. I was going to do it myself, but won't if I don't need too :-)

Post Reply