Shop Inventory Filling - Possible Coding Mistake

Discuss coding questions, pull requests, and implementation details.
User avatar
Rand
Posts: 68
Joined: Sat Nov 23, 2019 5:10 am
Location: Canada

Shop Inventory Filling - Possible Coding Mistake

Post by Rand » Tue Nov 26, 2019 5:04 pm

This is (slightly modified) from my post in the Shop Inventory Mod topic in Mods & Features General:

This is all regarding some code in DaggerfallLootDataTables.cs, specifically these lines:
49: public static byte[] itemGroupsArmorer = new byte[] { 0x02, 0x50, 0x03, 0x14 };
60: public static byte[] itemGroupsWeaponSmith = new byte[] { 0x02, 0x1E, 0x03, 0x46 };

I've noticed something. It looks like the arrays from DaggerfallLootDataTables.cs are read and processed in order (in pairs) from left to right by a function in DaggerfallLoot.cs that generates the shop inventory, and the second number of each hex pair is related to the appearance chance. This is modified slightly by the rarity (from ItemTemplates.txt). Weapons all seem to have the rarity 1 (meaning no change of chance to appear), while armors all see to be rarity 3. This seems to mean (based on line 194 of DaggerfallLoot.cs) that the armor's chance of appearing is 90% of the number in the array, so if it's 0x32, it's not 50%, it's 45%.

(I'm puzzled by the bits about shop quality and buildingData. I can't find where the numbers are being drawn from, and I have no idea of their magnitude.)

But wait, it gets worse. In standard Daggerfall Unity, armor shops use this array: { 0x02, 0x50, 0x03, 0x14 } (I think) this means check armor first, 50% chance (that turns into 45%), and if nothing, try for weapons at 20% chance. While weapon shops use this array: { 0x02, 0x1E, 0x03, 0x46 }. If I'm interpreting the code right from the DaggerfallLoot.cs code, this means that weapon shops check for armor FIRST (at 30%, lowered to 27%), and only if it doesn't pass, check for weapons at 70%. That means that the chance of a weapon is effectively 73% of 70%, or 51.1%. Lower than intended, and more armor than intended.

If I'm correct, the array for itemGroupsWeaponSmith should be reversed, from { 0x02, 0x1E, 0x03, 0x46 }, to { 0x03, 0x46, 0x02, 0x1E }, then the percentages of generation should match what was intended.

Maybe I should let one of the devs know and let them check my analysis for accuracy so if I'm right, they can fix it in the next build.

Can someone who knows what they're doing (i.e. not me ;) ) look at this and see if this is an (admittedly minor) issue?

User avatar
Interkarma
Posts: 5136
Joined: Sun Mar 22, 2015 1:51 am

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Interkarma » Tue Nov 26, 2019 8:43 pm

As far as I'm aware, Allofich and Hazelnut matched shop stocking to classic. The tables you reference come straight out of FALL.EXE - itemGroupsArmorer table is found at offset 1BAC51 and itemGroupsWeaponSmith is found at 1BACB0. The way these tables are processed into items should be a match for classic's process.

This doesn't rule out mistakes on our end of course, and I always appreciate feedback. I'll defer to Hazelnut here, as he was most involved in this system.

Shop quality and buildingData is combined from both MAPS.BSA and BLOCKS.BSA. The former has global information unique to each location, the latter has local information specific to a block. As a location is constructed at runtime, live building data from both sources is combined into a resultant set.

I'll move this down to developer discussion.

User avatar
Rand
Posts: 68
Joined: Sat Nov 23, 2019 5:10 am
Location: Canada

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Rand » Tue Nov 26, 2019 11:09 pm

Yeah, I found it in FALL.EXE, too. I checked there first to see if it was a Bethesda mistake.

I didn't think it was ever your guys' fault.
If it seemed like I was implying it was your possible coding mistake, I apologize. :oops:

I think this is the same sort of mistake that we see Bethesda makes in all of their games.

Another similar example of thoughtlessness from Daggerfall classic is the array (also originally from FALL.EXE) for determining random material quality (that makes Orcish items oddly rare):

ItemBuilder.cs, line 41:
static readonly byte[] materialsByModifier = { 64, 128, 10, 21, 13, 8, 5, 3, 2, 5 };

It should be clear that the 5 for Daedric (at the end) was added on after, or maybe raised to 5 from 1 during testing (to make it more common), without adjusting the other values properly in response.

Anyway, thanks for taking a look. :)

User avatar
Interkarma
Posts: 5136
Joined: Sun Mar 22, 2015 1:51 am

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Interkarma » Wed Nov 27, 2019 12:09 am

Thanks for the excellent feedback and possibly spotting a bug carried over from classic. We've fixed obvious bugs from classic elsewhere, so happy to do it here as well once everyone has arrived at a consensus solution. Looking forward to seeing where it goes. :)

User avatar
Ralzar
Posts: 826
Joined: Mon Oct 07, 2019 4:11 pm
Location: Norway

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Ralzar » Wed Nov 27, 2019 8:01 am

Both these do really seem like simple coding errors. How it should work seems pretty obvious. That Armour shops mighthave some weapons and Weapon shops might have some armor.

Also the Orc/Deadric ratio has allways just seemed off. Deadric starts cropping up weirdly often before you have even seen anything Orcish (or usually Ebony). And looking at the rest of the array makes it pretty easy to assume the original thought behind it: A bunch of Iron, Lots and lots of Steel, Silver is really rare, then from Elven the chances naturally decline until they hit Deadric which should be super rare.

This array would be really nice to be able to mod, because, if I'm not mistaken, you could use it to actually deactivate some material types from appearing in game (except for some particular circumstances I'm guessing). So you could decide to, for example, to have only Iron and Steel and have a really small chance of Mithril.
My released mods

"I feel like Ralzar specifically wrote those mods for me and then said "Use them". Just so he could watch me die more."
-FuzzyBeanPlays

User avatar
Hazelnut
Posts: 2012
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Hazelnut » Wed Nov 27, 2019 2:18 pm

It certainly is odd that daedric is more common than the two materials that precede it, but I am not convinced it's a bug or error. It might be, but I think like some other peculiarities it may have been intentional. I doubt we will ever know for sure. Personally I don't feel this is clear cut enough to be considered a bug that should be fixed in DFU, but I might be in a minority there.

This was actually one of the motivating factors for the new quest and armour service for FG members (and later to be extended to KOs) that I've developed for the Roleplay & Realism mod since it's really hard to complete an Orcish or Ebony set of armour.
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Hazelnut
Posts: 2012
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Hazelnut » Wed Nov 27, 2019 3:26 pm

Oh, and regarding the original post in this thread, all that stuff was done by Allofich reverse engineering classic. I did the DFU code that stocks the shelves and re-stocks, keeps track and allows purchasing but he did the generation of the items that go in the shelves based on classic code.

If there's a mistake from classic with evidence then I would be okay to change it since Allofich has been gone for a while now. Without evidence I would generally trust his work which was usually spot on. As you can see from the logic and byte arrays he basically reproduced the asm algorithms from classic code into c# versions in DFU.

Rand, I'm quite confused when you say "Lower than intended, and more armor than intended."... what or whos intention are you referring to here?
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Rand
Posts: 68
Joined: Sat Nov 23, 2019 5:10 am
Location: Canada

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Rand » Wed Nov 27, 2019 9:53 pm

Hazelnut wrote:
Wed Nov 27, 2019 3:26 pm
Rand, I'm quite confused when you say "Lower than intended, and more armor than intended."... what or whos intention are you referring to here?
Oh, it's pretty simple, though I could clearly have explained my thinking better.

The short answer is: whoever at Bethesda chose to have armor shops generate armor 80% of the time, and weapons shops generate weapons 70% of the time (on average, of course).

But first, bear in mind that's it's entirely possible that I may have misunderstood parts of the code that generates shop inventory. Before anything else, a programmer should definitely check my analysis. That's mainly what my original post was about, by the way. :oops:

As I said, I believe the code checks the items by category from beginning to end, one pair of numbers at a time. For Armor shops, it first sees the 0x02 0x50, recognizes that it's a check for armor and "rolls" for the chance to fill a slot (and since armor has rarity 3, the math from the code makes the initial 80% become effectively 72%). Then, if the check fails, it checks again with the next set of numbers, 0x03 0x14, for weapons at 20% (since rarity 1 doesn't alter the calculation from the chance number).

The problem is the weapon shops (weaponsmiths). They, too, check armor first (0x02 0x1E) at 30% (lowered to 27% due to the rarity), and only then if it fails does it check for weapons. The weapons numbers are 0x03 0x46. I believe that the initial intent was to have a 70% chance (46 hex) of weapons (like the armorer has an 80% (actually 72%) chance of armor), but due to the armor check going first, and necessitating it to fail before weapons are checked, the effective percentage is lowered to 51.1%, meaning that fewer weapons are being generated to fill shop slots than the intended 70% set by the original person who chose the number, with a corresponding increase in armor available in the weapons shops.

I think it's just a boneheaded entry error made by Bethesda that has gone unnoticed until now. The reason is, I think, an alphabetical/hexadecimal ascending sort in an Excel table all those years ago, because in all the lists the items group codes are all in ascending value, with the exception of the last one: itemGroupsTemple. Fortunately, it doesn't negatively affect any other shop in any significant way, with the possible exception of itemGroupsAlchemist, which, oddly, checks for gems first (and due to the existence of dedicated gem and jewellery shops, I would make gems last for alchemists, thus making other ingredients slightly more common).

I hope that clears it up! :)
Last edited by Rand on Wed Nov 27, 2019 10:12 pm, edited 2 times in total.

User avatar
Rand
Posts: 68
Joined: Sat Nov 23, 2019 5:10 am
Location: Canada

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Rand » Wed Nov 27, 2019 10:10 pm

Hazelnut wrote:
Wed Nov 27, 2019 2:18 pm
It certainly is odd that daedric is more common than the two materials that precede it, but I am not convinced it's a bug or error. It might be, but I think like some other peculiarities it may have been intentional. I doubt we will ever know for sure. Personally I don't feel this is clear cut enough to be considered a bug that should be fixed in DFU, but I might be in a minority there.

This was actually one of the motivating factors for the new quest and armour service for FG members (and later to be extended to KOs) that I've developed for the Roleplay & Realism mod since it's really hard to complete an Orcish or Ebony set of armour.
gimble's analysis convinced me that it's basically a bug:
viewtopic.php?f=5&t=2656&start=10#p31680

But the question is: is it policy to stay as true to the original as possible?
My vote would be an unqualified yes, and so to leave it as-is in the default code, but also to expose the numbers to allow mods to rebalance them.

To this end, I already set up a spreadsheet to work the numbers myself. It's not too hard to find ones that give more reasonable results, differentiating materials by level more evenly, and moving Daedric and such to the higher levels they belong at in a levelled world.

User avatar
Ralzar
Posts: 826
Joined: Mon Oct 07, 2019 4:11 pm
Location: Norway

Re: Shop Inventory Filling - Possible Coding Mistake

Post by Ralzar » Thu Nov 28, 2019 10:21 am

Rand wrote:
Wed Nov 27, 2019 10:10 pm
To this end, I already set up a spreadsheet to work the numbers myself. It's not too hard to find ones that give more reasonable results, differentiating materials by level more evenly, and moving Daedric and such to the higher levels they belong at in a levelled world.
And maybe have shop quality affect selection? If we're talking about modding this.
My released mods

"I feel like Ralzar specifically wrote those mods for me and then said "Use them". Just so he could watch me die more."
-FuzzyBeanPlays

Post Reply