Boat Buzz Date Issues

by **Bill Basler** » Thu Jan 20, 2011 5:49 pm

All, we have been working on some things here on Boat Buzz in preparation to making some significant enhancements to the forum. More about this later.

We have been trying to repair a few crashed database tables. Good news is we have repaired the tables in some ways. For example, some of you mentioned that you could not get proper results on "Mark All Posts as Read" on the Hull Construction forum. This has been fixed.

Unfortunately our fixes seem to have reset all of the posting dates to 1969. We will be trying to chase down a fix for this later this evening. Either a repair is in order, or we will revert our recent repairs and try again, to repair the tables without resetting the dates.

I will pull Boat Buzz down about 10:00 p.m. I know this is "prime time" but we have to get the SQL tables restored to get 100% stability back.

Please don't post anything too important between now and tomorrow morning in case we need to revert our changes back to last evening!

by **Bill Basler** » Thu Jan 20, 2011 5:57 pm

On second thought, we may just let this ride. The repair of the tables was successful Only the dates at the very top level of Boat Buzz seem to have been time shifted to 1969. ONce within any of the topic areas the individual posts have retained their proper dates.

This is a minor annoyance that may affect your email digests, or your top level browsing...but the dates will correct themselves as a post is added or edited within each topic area.

by **cmeke0870** » Thu Jan 20, 2011 11:00 pm

All my recent posts and replies seem to be missing

by **Bill Basler** » Fri Jan 21, 2011 1:29 am

Chuck, can you give me an idea of when the posts were?

by **cmeke0870** » Fri Jan 21, 2011 8:50 am

Appears to be everything over the past two weeks. I don't have any specfic dates. I had new posts in Finishing and Hardware and I had updated my Hull Construction post. It wouldn't be hard to put the any singificant info back up.

by **cmeke0870** » Fri Jan 21, 2011 8:55 am

For example Bill, this thread had 4 or 5 new replies that are now missing

http://www.chris-craft.org/discussion/v ... php?t=4969

by **jahearne** » Fri Jan 21, 2011 1:42 pm

Hi Bill,

If this happens again, I work for data recovery company. I specailize in recovering from complete system failure of RAID arrays, but there's no magic in recovering from a corrupted database other than restoring from backup as long as the backups are good. DBA's are worth their wieght in varnished wood when databases die.

I also have experience in disaster recovery planning and business continuity best practises. I'd be more than happy to discuss with you how to prevent or minimize problems like this in the future.

Regards,
John

by **Bill Basler** » Fri Jan 21, 2011 2:07 pm

Thanks John. I hear what you are saying. We have a good backup scheme of nighty, weekly, and monthly backups run in rotation with full and incremental backups. These are taken off site, and in fact some of the backups take place over wire to a remote location. Boat Buzz runs ons a large Raid array, Raid Level 5.

The trick with these forums is the changes happen 24/7 with the database constantly getting revised. If we were to return to one of our backups, best case scenario would have been the prior evening's incremental, and in this case the corruption happened just before the next incremental backup.

This actually happened a few months back. The forum has been stable and working fine, but in the cases of searching for specific words, such as "boat" which is mentioned in thousands of posts, the user was greeted with a SQL table crash error. We seem to have gotten this resolved and we may have lost a few posts in the process, but we are looking into bring specific ones back.

I'll let you know how we are doing. By the way, there is back story to all of this as well, in that I am upgrading Boat Buzz to a newer version of phpBB and we are working on some very custom features.

It's good to know you are out there, as crashes give me gray hair. Good RAID systems and backups cover our rear most of the time, but there are exceptions.

by **Bill Basler** » Fri Jan 21, 2011 2:13 pm

Remind me John, what is a DBA?

by **mattbyrne** » Fri Jan 21, 2011 3:15 pm

Bill,

DataBase Administrator

by **Bill Basler** » Fri Jan 21, 2011 4:18 pm

Got it. When I have to deal with my own database issues, am I one of those? :wink:

by **jahearne** » Fri Jan 21, 2011 4:52 pm

Bill,

You have a very good backup scheme; it would be hard to one better that not to mention very costly.

I can't say enough for offsite backups. Are your onsite backups going to a different system? You don't want your backups going to the same RAID array or hard drives as the data you're trying to backup. If there is a hardware failure there goes your data and your backups too.

Most RAID array hard drives come from the same manufacturer, the same lot, same revision, firmware, etc. If one hard drive fails, you can bet that another one will be close behind it. With a RAID-5, by the time you get a replacement drive and it rebuilds depending on the size it can take all day to rebuild a 1 terabyte drive especially if you have to wait for a replacement drive to arrive. A hot spare is good idea as well has having an extra hard drive on hand.

Maintaining something a dynamic as the database the supports this forum is a lot of work not to mention skill. Besides the wonderful publication of the Brass Bell, you get my thanks and respect for maintaining this great forum. I look forward to the new features and if I can be of help, please don't hesitate to contact me.

Thanks,
John

by **jahearne** » Fri Jan 21, 2011 4:57 pm

Bill Basler wrote:Got it. When I have to deal with my own database issues, am I one of those?

Wear many hats...

by **Al Benton** » Fri Jan 21, 2011 5:06 pm

Just glancing through some of the subjects it appears that most of them lost posts and new topics for some of December and all of January up until yesterday, when new ones were being posted.

For instance the "Uh-Oh" topic in Hull Construction is missing many posts between Nov. 2010 and yesterday.

Al

by **Bill Basler** » Fri Jan 21, 2011 5:21 pm

John, I have to admit that managing databases..as well as protecting them, gives me gas! I really worry about these things, as when they go down, it creates chaos.

We actually have our system drives mirroring to other internal drives on the same server. These are hot, and in the case of minor problems, we can drag files back in place from backups pretty easy. We also back up all of our system, php and Apache files to a removable drive that creates bit-for-bit copies of the entire volume. These are actually bootable volumes if we need to make one of them a boot drive and get up and running.

These are normal drives outside of the storage Raid. Our RAID is a level 5 mirroring to an entirely different, but identical RAID 5 array.

I have been dong this since our first file server was an 8-foot rack filled full of Seagate Barracuda 500 Mb drives....faulty ones at that. After about 9 months they started failing one after another. At one point the failures were happening fast enough that even a single parity drive was not enough. We couldn't get replacement drives fast enough. I can't remember the costs (showing my age here) but they were price enough that we didn't keep any more than we needed on hand. You're right though, to swap a 500 Mb drive and do a rebuild was a pretty simple matter...much less time than a 1.5 or 2 TB drive. Seagate finally acknowledged the issue, and paid for the replacement of all of our drives for 1.5 GB drives We thought we had died and gone to heaven!

Currently our RAID is somewhere around 80TB. Getting this in my pocket for an off site is getting more challenging. We are now backing up to hard drives as there is not as much of a cost advantage to tapes as there once was. We have used DAT, DLT, AIT AIT II AIT III, and most recently with an AIT III jukebox loading barcoded tapes. Expensive and really a pain.

Hard drives are much easier. We are looking for methods to backup entire over the web to another identical server lying in a different building...different city for that matter. Unfortunately this takes huge bandwidth, but we're getting closer to getting this done.

Thanks a bunch for your input.

by **jahearne** » Fri Jan 21, 2011 6:19 pm

Wow, 80 TBs! I had no idea.

Let me rephrase that from "very good" to "most awesome". The one thing that drives me crazy about mirrors is if one mirror gets corrupted then the other mirror goes bad too. I haven't played around with it much, but have you looked into shadow copy or snapshots? For backups over the web, will you be doing a hot site or something like it?

by **Bill Basler** » Sun Jan 23, 2011 12:47 pm

OK, we are making progress on the repair of the crashed SQL tables. For now, please be aware that the repairs are complete, but we have lost some posts between December and current.

The main thing we are after is absolute stability of the SQL tables. We seem to have that now. We know what exact topics are missing and we will be working to restore those in the fairly near future. If there is anything ultra important that you can repost without much work, you can do so. Otherwise, we will be working through the missing posts, making sure we don't bring any corrupted data back in in the process.

by **Bill Basler** » Sun Jan 23, 2011 12:53 pm

There appear to be 136 topics that were affected. We know exactly which ones. We will be bringing these back online as we work through them.

Boat Buzz Date Issues

Boat Buzz Date Issues

Who is online