Apologies to our members, our site was down for 2 and a half hours tonight from around 7.30pm through to 10pm. That’s a killer for us because tonight was shaping up quite nicely. It wasn’t me guv .. I wasn’t messing around with our server. Unfortunately our host had a network outage. It seems some gremlins have crept in with a recent equipment upgrade. We were off the air yesterday as well for close to 45 minutes.

Being reliant on network connectivity is enough to give anyone grey hairs, and these outages aren’t pleasant. Yet, despite our tech base we’ve been remarkably trouble free. Knock on wood!

And don’t forget .. even if our site happens to be offline our shop will still be OPEN and the music ON. Come on down and we’ll do it old school. All manual based and you can even checkout the covers in our window. Andrew was manning the shop tonight and caught the short straw as the system came up just as he was about to shut up shop .. dang back processing those manual orders. Sorry dude!

What happens when you can’t contact your co-lo provider, via email, 1300 voice mail, control panel ticket, and then find his mobile doesn’t deliver a dial tone?

You start to worry.

Servers aren’t exactly perpetual motion machines. They freeze up if someone like me messes too much with them. Hardware failures do occur, memory needs to be added, disc, power supplies can fail. If any of that happens, I need access ASAP to fix it, and when I can’t raise my colo company AT ALL, it’s time to get out. And it’s been an intense few days getting out.

After much calling, I raise the account manager who deals with my colo reseller*, and find they can’t get him either, and have placed his account in dispute. If it goes legal, the cabinet gets locked and my server with it. Bye bye server. DANG. This is getting seriouser by the minute. Ok. more calls, and finally get a response from my colo reseller and yep - I can pull my server. Phew. Sourcing another HP dl385 2xdual core Opteron, 8Gig RAM, SAS RAID ain’t exactly a 7-11 pickup (even though this isn’t a high end machine by any account). From ordering to setup I reckon is prob 4-5 days at best.

So, server comes out at 10.00am Sunday morning - where’s it going to go? Full thanks to Chris at Intervolve for not breaking a sweat and getting me booked in for 10.30am Sunday. The story doesn’t end there of course. Nothing ever goes smoothly, and with the machine all hooked up I then discover I had problems seeing the IP addresses I use for my shop systems. So - back down to manual checkouts (people could still find and rent DVDs) - and with my head stuck in an ssh session trying all sorts of re-config options, and still chatting happily to customers and demoing the system - what a day! Thanks again to Chris - he stepped in later and dealt with the problem and so all is now good. If it doesn’t kill ya it’ll make you stronger as they say!

* For the non-techs out there (bless you all :-), co-location is when I physically place my server in someone else’s data centre facilities, which offers temperature control, backup power, secure access, internet connectivity, fire fighting kit and so on. In my case, I use a reseller of data centre space, as the big data centres don’t want to deal with a small player like me.

My apologies to anyone who has tried to access APPLEBOX in the last 26 hours. My hosting provider had a serious network problem, and was unable to stably provide network access to the APPLEBOX server (the server itself has been operating fine). I will update this post once I find out what went wrong.

UPDATE 14/5
Ok - I have discovered network outages are not much fun. The problem was administrative (on the host provider’s behalf) and not equipment or mis-configuration. There has been some compensation, but the choice left for me is to stay or move. As if there wasn’t enough to do already …

UPDATE 16/5
Another outage (couple of hours this time) as my server gets moved to a new data centre. All back up and running and performance is great. Just clocked a 10Mbps download so no complaints there.