Page 1 of 1

Recent Downtime

Posted: Sun Mar 05, 2017 1:20 am
by Interkarma
It has been a frustrating few days after a recent AWS outage borked MySQL connections for our blog and forums. The host resolved this quickly for sites hosted in North America region, but dfworkshop.net was hosted in the closest datacentre to Australia (Asia region), and this continues to be down at time of writing. And after almost 48 hours, I’m still unable to obtain time-frame for fix.

I’ve worked around this by migrating sites to North America region with functioning MySQL servers. Unfortunately, databases couldn’t migrate because source servers still aren’t live. To fix this, I restored from the most recent backup available taken on 25/02/17. For the blog this isn’t a problem as nothing was posted in this time, but for forums this means all posts, replies, PMs, and new user registrations created after 25/02/17 are not part of the restore.

I’ve locked the forums for now and will continue working with host to try and recover a later version of databases. Outcome is dependent on how quickly host can resolve MySQL issue in Asia region. If they can’t fix this in next 24 hours, I’ll unlock forums and we’ll just move on. I’d rather have a functioning site than sweat over a few days of lost posts.

I apologise if you registered to forums or posted anything after the 25th. Forums will be back online within 24 hours and we can pick up where left off one way or another. Thank you for your patience!

Update: Forums are back online now. Unfortunately it doesn't look like I'm getting those live databases back. This means I have to start from most recent backup as of 25/02/17. Please repost to any active conversations you were participating in prior to outage and we'll be back on track in no time.

Re: Recent Downtime

Posted: Sun Mar 05, 2017 4:31 am
by Arl
This is how I was when I loged in today:
Spoiler!
Image
I'm exagerating, we'll tune in again in no time.

Re: Recent Downtime

Posted: Sun Mar 05, 2017 5:13 am
by Interkarma
Haha yep, I felt much the same. :)

At least it's less than a week total lost (excluding downtime when nobody could post anyway). I'm just glad I keep fairly regular snapshots of everything - although in this case a bit more regular would have been helpful. Timing was about worst-case with failure happening day before next snapshot would have happened. Oh well, we're just making Daggerfall here, not saving lives.

Re: Recent Downtime

Posted: Sun Mar 05, 2017 6:25 am
by LypyL
It sucks we lost everything during that time frame, but it's great to have everything working again regardless. Thanks for all your hard work! :)

Re: Recent Downtime

Posted: Sun Mar 05, 2017 1:14 pm
by TheLacus
LypyL wrote:It sucks we lost everything during that time frame, but it's great to have everything working again regardless. Thanks for all your hard work! :)
+1 :)

I'm trying to restore what i can from the cache of my browser, i guess it's better than nothing.

Re: Recent Downtime

Posted: Sun Mar 05, 2017 1:36 pm
by Interkarma
Nice one TheLacus! Thank you for posting what you could. :)

Re: Recent Downtime

Posted: Sun Mar 05, 2017 11:58 pm
by Arl
Good job in your restoring posting TheLacus, Thank you!

Re: Recent Downtime

Posted: Mon Mar 06, 2017 7:05 pm
by Narf the Mouse
I would strongly advise switching to a host that doesn't use MySQL. The "why" of it can be summed up like this (and is about what I remember from a long lecture on database systems):

A proper database system ensures that the data on the disk is always readable and consistent. That means that what got fully written last is still on the disk, and that what you read is what you wrote. In addition, even if the last write didn't complete, or a disaster occurred, the database is still readable, and still contains everything up to the last write that did work.

In short, with a proper database system, the only way to actually ruin the data is to physically ruin the disk (which includes running out of viable writes on an SSD).

MySQL guarantees none of that. Its only advantage is price.

Re: Recent Downtime

Posted: Tue Mar 07, 2017 6:48 am
by Lingering_Trees_Gabe
Ah, I've had this happen before, back when my wife and I ran a phpBB forums. I used to download the forums to my pc once in a while to make sure nothing was ever lost, and that actually helped, but we were a small forums, without much on it, so the files were small. I vaguely remember something in the ACP that allowed you to send the file somewhere else, like another online storage host.

I've never trusted online file storage though. Not a huge loss either, so it's no big deal. I hate mySQL, always seemed unreliable.

I do rather enjoy the theme you have installed btw. For someone like me who suffers from chronic health issues and gets headaches a ton more than I should, the contrast makes this easy to read and look at! So for that, I must thank you! :mrgreen:

On a side note, and rather frivolous, ever though of adding Daggerfall Smilies?