Forum Moderators: phranque
The larger dedicated server provider in the world, I am sure millions of 404s every second from 100s of thousands of sites, this will undoubtedly be a test to see how quickly they can respond to the situation and get servers back online. it has been few hours so fare!
[newsblaze.com...]
[edited by: Brett_Tabke at 11:34 am (utc) on June 1, 2008]
[edit reason] added link [/edit]
I was unlucky/foolish enough to have both DNS servers on the same machine, so my sites are totally down (not timing out) and email is bounced back instantly (not queued).
consider adding a free dns nameserver to your nameserver list (at the domain registrar). sign up for free dns and point it to the correct IP, so when things come back online, the nameserver will work the way you want it to.
They seem to be getting a handle on it...
It also seems that the outgoing traffic had serious package losses to some parts of the world. Mail delivery connects to norway took over 10 seconds, so our mail delivery timed out.
All in all: nothing serious, just annoying for us, but some additional fail-over setup seems to be wise!
P!
Never blame the host, most big hosts do a really good job at infrastructure but there's always the unexpected, including natural disasters, no matter how well you plan and design your data center.Now you know you need a contingency plan.
It doesn't cost much to have a lesser powered backup server on standby with a carbon copy of your site ready to go live at a moments notice. It can even be kept up-to-date using a cron job with RSync (on Linux) so you've only lost an hours worth of data (orders) worse case. I also keep a daily backup locally just in case all of my best laid plans fail simultaneously, I can still upload to a new server elsewhere from scratch.
Besides, there are many other reasons to do this such as complete hardware failure, getting hacked, having major routers go down outside the data center, etc.
A few years ago a main router at Level 3 blew out in the Los Angeles region and it cut off a large chunk of the state and it took 6 hours to restore because there was only 1 backup part and it was a 3 hour drive away (and back) and for some odd reason there was nobody where the part was located that could drive it to LA, sounds stupid I know, but that's how big business operates.
So having a backup server in another part of the country is generally a good idea.
While you're at it, put your server on a 3rd party dynamic DNS service so you can move your site from server to server in mere seconds.
All great points IncrediBILL. I'm taking this all in as a learning experience. We were being very naive to not have a rock-solid disaster plan in place.
it is a good idea to spread them (and their DNS) between at least two independent hosting companies in locations geographically far apart.
Most people don't realize it but the DNS server is the single point of weakness in most data centers and it was pointed out quite dramatically by a botnet attack against a security company a few years back. The botnet couldn't bring the site down with all their security so they stopped attacking the site itself and attacked the hosts DNS servers with a flood of traffic instead and brought EVERYONE being hosted there down, including the intended target.
So more than one DNS server from more than one source is a really good idea.
There's always going to be a worse case, and it appears that The Planet has provided the many with one.
Ours has always been to colo or lease across the country. With switching at a premium these days, it's still felt that it's worth it to stay up, "always". You can never have too much redundancy, especially when dealing with network wide outages, or even outages across multiple networks.
One might go offline in Virginia, and at the second it does, L.A. picks it up without so much as even a blink.
Providing yourself with a backup, or some other kind of redundancy plan is a very good idea. Don't throw it all into just one DC or Network if you can help it. Spread your data out a bit.
I have been on both sides of the planning spectrum - from a ridiculously complicated backup plan involving such steps as keeping three backup servers with as near-time data as possible and then hand-transporting them separately to three previously-chosen locations to get things up and running again (I swear) to NO plan (like, uh, right now with my personal site). Somewhere in the middle would probably be a good idea.
I'm still not up, and I am about to end up whimpering incoherently in the fetal position near my computer.
Keyboard, meet forehead.
I'm disappointed ThePlanet has decided to let things happen like this. There was most likely overloading in the original RackShack/EV1Servers datacentre in Houston.. I feel they've been giving little attention to the legacy customers and datacentre.
I'm disappointed ThePlanet has decided to let things happen like this.
You think someone just woke up and said "Hey! Wouldn't it be fun to blow up a transformer today?"
I kind of doubt that.
You think someone just woke up and said "Hey! Wouldn't it be fun to blow up a transformer today?"
No, of course not.
I've just been let down since EV1 ceased to exist. If there was proper maintenance, I'm sure such an explosion on that large of a scale would never have happened.
No biggie for me, luckily...nothing but a few hobby sites there.
Hope they get things back up quickly for you guys that are losing money over this...at least they appear to be trying.
[edited by: engine at 12:03 pm (utc) on June 3, 2008]
It doesn't cost much to have a lesser powered backup server on standby with a carbon copy of your site ready to go live at a moments notice.
Good advice.
Reset DNS then add the site to the new server and upload files db's etc while dns is busy changing, it's quite quick these days DEPENDING on ISP I have one customer who STILL is down and a ping retunrs the OLD server (supanet UK) anyone on BT should see the changes within an hour usually
YMMV
edit: my reply was to martinibuster who asked about DNS propogation times, looks like he edited his post
It doesn't cost much to have a lesser powered backup server on standby with a carbon copy of your site ready to go live at a moments notice.
So how do you deal with the DNS propagation? I have a complete backup of my db and files and have another server out of theplanet and I was thinking of switching the DNS when I first learned what has happened, but I was also worried that by the time it propagates the planet might have fixed the problem already and have to switch back again. I thought it will just prolong the downtime of the site. So just decided to wait and do some other things.
It's up to you to decide ... and balance your needs between the ability to move fast or the lighter load on the DNS servers.
Having more DNS servers can relieve your server load if it ever becomes a problem, but do note this performance is also visible to end users as a cached answer gets to them significantly faster than a query that needs to be forwarded.
Moving your authoritative nameservers about takes as long as the TLD owner has set his TTL values on your NS and (optionally) glue records. That's out of your control, so you need to have them spread out enough not to run into trouble with loosing all of them.
Should you loose the master server: make sure to promote one of the remaining slaves to master and reconfigure the other slaves. I you don't, the slaves will eventually stop handing out information as it grows stale.
All those timings are configured in the SOA DNS record. But individual records can have their own TTL.
I feel very fortunate not to be affected, but I am still concerned why a Halon system wasn't present, or if it failed.
Bye, Julio.
I have to say I feel better about TP today than I did a week ago - crazy huh?
Bill
the Webmaster General forum Charter [webmasterworld.com] specifically prohibits discussion about hosting providers.
sites hosted there are still down
I'm really surprised the mainstream media (USAToday tech section, etc.) isn't picking up on this.
I'm really surprised the mainstream media (USAToday tech section, etc.) isn't picking up on this.
I'm surprised at this also. I get daily newsletters from a couple tech publications and no mention of it. The only place I've seen it mentioned at all is in forums. You'd think with @ a million sites down at one time it would have received some attention.
