Index was outside the bounds of an array

Discuss modding questions and implementation details.
User avatar
Magicono43
Posts: 1139
Joined: Tue Nov 06, 2018 7:06 am

Re: Index was outside the bounds of an array

Post by Magicono43 »

Just as another note, this was the post I made related to this before and some of the testing we did involving this, might give some more context: viewtopic.php?p=45851#p45891

The problem we are sort of assuming is possibly a limitation with the MCS compiler, and possibly a length limit causing problems during the mod loading process, for .dfmod "built" mods. As none of this happens with virtual mods, which I believe circumvent this entire process of putting the entire mod code into a string and putting that string into an array index.

User avatar
Hazelnut
Posts: 3015
Joined: Sat Aug 26, 2017 2:46 pm
Contact:

Re: Index was outside the bounds of an array

Post by Hazelnut »

I should add my perspective on it in case it's useful.

So firstly, this is coming directly from code inside the MCS compiler, TheLacus identified the line here and I independently reached the exact same line of code. (wasn't double checking TheLacus, just hadn't read his post until after - silly me)

It's not clear how this line is giving the index OOB but the very interesting aspect is that the problem mods have very large classes of 1900+ lines which is why I suspected reaching some limit of parsing buffers or something like that. However, the other really crucial bit of info (which didn't strike me when investigating with Ral & Magic earlier today) is that these mods compile just fine, but the compiler breaks on the next mod it tries to compile. That is why the ordering of mods makes a difference, and placing one of the problem packages at the end of load order makes the issue disappear.

That explains why which mod exhibits these problems has been so confusing, they seem to be causing a side-effect that knocks on to other mod parsing and order is important. I did briefly try to instantiate the compiler for each mod, but it made no difference. It's a puzzler for sure this one.

We know there's a definite solution which TheLacus is working out so the code can be precompiled into a dll, so that's something at least, but it's frustrating not to be able to figure out what's happening here.
See my mod code for examples of how to change various aspects of DFU: https://github.com/ajrb/dfunity-mods

User avatar
Interkarma
Posts: 7236
Joined: Sun Mar 22, 2015 1:51 am

Re: Index was outside the bounds of an array

Post by Interkarma »

Thanks for additional context everyone. Yeah, if we're just now hitting some hard limit inside of the MCS compiler that would be... harder to fix. It means the core issue has been there from the start, and we're only seeing it now mods have grown to a certain size and complexity. :(

User avatar
Magicono43
Posts: 1139
Joined: Tue Nov 06, 2018 7:06 am

Re: Index was outside the bounds of an array

Post by Magicono43 »

Or an excessive amount of bloat and ineffective coding practice ;)

Guilty as charged here, lol.

User avatar
Ralzar
Posts: 2211
Joined: Mon Oct 07, 2019 4:11 pm
Location: Norway

Re: Index was outside the bounds of an array

Post by Ralzar »

Magicono43 wrote: Sun Jul 19, 2020 9:41 pm Or an excessive amount of bloat and ineffective coding practice ;)
I resemble that remark!

User avatar
Interkarma
Posts: 7236
Joined: Sun Mar 22, 2015 1:51 am

Re: Index was outside the bounds of an array

Post by Interkarma »

Had a quick look before work - and yeah there does appear to be a potential issue with the maximum LineOffset delta from last checkpoint. I made a more detailed post in other topic here.

viewtopic.php?f=12&t=3929&p=45917#p45917

User avatar
DigitalMonk
Posts: 111
Joined: Sun Nov 10, 2019 8:01 pm

Re: Index was outside the bounds of an array

Post by DigitalMonk »

(Apologies beforehand if this is useless/obvious/already-considered-and-discarded :roll: )
Hazelnut wrote: Sun Jul 19, 2020 9:17 pm It's not clear how this line is giving the index OOB

Code: Select all

			
// TODO: For eval only, need better handling of empty
int file_index = file == null ? 0 : file.Index;

// FIXME: This value is certainly wrong but what was the intension
int /*--->*/max = checkpoint_index < 10 ?
	checkpoint_index : 10; /*<---*/
for (int i = 0; i < max; i++) {
	int offset = /*--->*/checkpoints [checkpoint_index - i]/*<---*/.LineOffset; <-- line 265
	delta = row - offset;
	if (delta >= 0 &&
		delta < (1 << line_delta_bits) &&
		checkpoints[checkpoint_index - i].File == file_index) {
		target = checkpoint_index - i;
		break;
	}
}
The above makes sense when checkpoint_index < 10, where it processes the small number of checkpoints in reverse order. But if, for instance, checkpoint_index = 50, then max==10 and the indexing tries to access checkpoints[50-i], which will be indices 50, 49, 48, ..., 42, 41, reverse indexing the last ten checkpoints. That seems very strange. I suspect that it might need to be checkpoints[max-i], which would index the first "up to 10" checkpoints in reverse order. But I don't know what checkpoints actually is, or which end needs to be processed, etc. I'm just going by the obvious confusion in the comments, and thinking that this might be a refactoring error. Maybe max didn't originally exist and all the processing was done directly off of checkpoint_index. But then it was discovered that at least 10 checkpoints must be processed, so max was added to handle that, but they forgot to change it down on the indexing line.
Hazelnut wrote: Sun Jul 19, 2020 9:17 pm ... these mods compile just fine, but the compiler breaks on the next mod it tries to compile. That is why the ordering of mods makes a difference, and placing one of the problem packages at the end of load order makes the issue disappear.

.. they seem to be causing a side-effect that knocks on to other mod parsing and order is important. I did briefly try to instantiate the compiler for each mod, but it made no difference.
Sounds very much like a buffer overrun, as you suspected. The codes writes outside one of the working arrays/strings and writes over data in the heap from other components, and now they are in an invalid state. Couple of thoughts that might give ideas for quick things to scan for inside the compiler:
  • Corrupting other data explains why it doesn't fail on the problematic mod, but on what comes next. Most likely, nothing else is going on in the system while the compiler crunches, so it can get back to the information that it erroneously wrote on other components' memory, but once it exits, some of those components are effectively broken. I suspect this is true even if the problematic mod is loaded last -- the compiler doesn't run again so it can't notice the problem, but some random piece of system state was corrupted, which could cause extremely gremlin-esque issues.
  • In theory, C# shouldn't let you do a buffer overrun in normal code, but if the compiler has some unsafe code for faster processing, that code needs to be examined very carefully. Or just remove the unsafe flag and fix the code to work in the normal, runtime validated way. (search for unsafe)
  • Re-instantiating a new compiler object wouldn't change any static resources that the class uses. Those are allocated when the first object is instantiated and stay until the program ends (generally -- sufficient garbage collection might force them out), and the next instantiation would then inherit the damaged static. (search for static)

User avatar
Ralzar
Posts: 2211
Joined: Mon Oct 07, 2019 4:11 pm
Location: Norway

Re: Index was outside the bounds of an array

Post by Ralzar »

I've lost track of where we discussed this last :D

What's the situation with this now? Is the new mod packing/building tool up and running?

User avatar
Magicono43
Posts: 1139
Joined: Tue Nov 06, 2018 7:06 am

Re: Index was outside the bounds of an array

Post by Magicono43 »

So after updating my mods for 10.26, I used the pre-compiler option when building all of my mods, and after a little bit of testing it seems to have solved the "Outside the bounds of an array" issue that was happening, so it looks like the pre-compiling has worked as intended. Also feels like the mods build much faster with this option enabled. So good job TheLacus! Thanks for the work on making this so easy to use!

User avatar
Ralzar
Posts: 2211
Joined: Mon Oct 07, 2019 4:11 pm
Location: Norway

Re: Index was outside the bounds of an array

Post by Ralzar »

Yeah I'll second that. The pre compiler works really smoothly.

Post Reply