Forum Moderators: phranque
Background:
I'm planning on hosting my website at home on my ADSL line with dyndns.com providing dynamic to static translation! Dedicated bandwidth is expensive in my country and ADSL is the way to go. There are fairly cost effective hosts providing Static IP hosting on ADSL, but it's still much more expensive (about 3-4 times more) than doing it yourself. Also, I'm planning on eventually Load Balancing multiple ADSL lines when my website grows bigger (either with a load balancing router or iptables/pf solution).
The situation:
Google comes to my website every day now (MSN and Yahoo about every other day), and I have about 4-5 thousand pages in the Google index. I'm top ranked for many keywords in my country and don't want to loose a single page (yeah right!) in the index when I host it at home.
My question is:
How will hosting my website at home affect my rankings and search index? Because every 12-24 hours my ADSL line could be dropped and a new IP address assigned. dyndns.com will update about 2-10 minutes later with the new address! But the Google servers might have an outdated DNS entry when they start the crawl?
What will happen if Google is on the site at the time the line drops?
Do you think Google (googlebot) will still try the old IP address for a few days (could it start caching someone else's site or reading their content because someone else might have my old IP?).
Do you think some pages will dissapear because of the switch over (in the perfect world where there is no down time!)?
I can't see any pros AFAIK for the search engines for me to host it at home, but what are the negative side effects you can think of?
Are there things I could do to minimize the effect?
For example, could I determine how much time is left before I get disconnected (I could work that out or profile it for a best guess) and somehow limit the googlebot to index for that time frame etc.
Has anyone done this before and what were your experiences as far as search engines is concerned?
Looking forward to your suggestions, advise and insight!
it's still much more expensive (about 3-4 times more) than doing it yourself
Shop around hosting is a very competitive business. Consider offshore hosting. Doing it yourself can be interesting but is time consuming and costly to do well.
Consider
- upload speed of your ADSL connection
- uptime commitment of your ISP
- backup power supply
- security maintainance of all software involved
- are you there 24/7
- time spent on hosting duties = less time developing site
SEs don't like it when they can't get to your site. (Oh yeah, users don't like it either. :) ) We've had experiences with our servers being down for several hours (once because of a power outage at our co-location company, another because our switch died), and we took a big hit in Google a few days later (hundreds of thousands of pages dropped from the index). They cam back a few days after that, and we can't say for certain that the downtime directly caused the drop, but...
I assume that you're making money from your site (ads, affiliate programs, etc.)? Besides the SE perspective, when your site's down, it's not making you any money.
If you insist of hosting at home, at least get a static IP address. But if you're even halfway serious about your site, don't host at home with DSL.
Since this seems to be a concern of yours, I have to assume that your audience is local. OK, fair enough. But work out the numbers and consider the physics.
I see from your profile you are in South Africa. Before I checked that, my first thought was to suggest that you host in some other nearby country. But I imagine there are no good nearby choices, and S. Africa probaby has about the best connectivity on the continent.
How is your access to U.S.-based websites? European ones? How about this one? Is there a noticible delay vs. accessing local sites?
Consider hosting in the U.S. or Europe. Heck, you can get *10* U.S.-based hosting accounts for the price of *1* (commercial) DSL line in the U.S. For us, it would be absolutely foolish to host a site on one's own DSL line.
That's an important point. Read your ISP's terms of service. Most don't permit hosting a website on a DSL line meant for personal, home use. If you want to host commercial traffic, you need to enter into a different agreement.
As an example, in the U.S. a personal DSL line can cost as little as $20/month. "Premimum" service might cost as much as $50/month. If you want to use it for commercial traffic (a website, a company branch office, etc.) you are looking at more like $100-$150/month. Note that you normally will get a static IP address with commercial service, so I have to assume you are considering using "personal" service.
Don't ignore this. You could get yourself in a tight spot if you ignore your providers TOS and there is little or no competition. You could be stuck with no service at all if they catch you.
The tradeoff for offshore hosting is that the packets will take longer to go back and forth, possibly causing a noticible delay. On the other hand, you have oodles of cheap bandwidth available, at least from your host to the net. The one may well compensate for the other.
Of course, there could be a nasty choke-point getting to S. Africa. I'd suggest reviewing a good bandwidth connectivity map (any suggestions where to look?) to see just where S. Africa has the best connectivity to. But I'd have to guess that the U.S., western Europe, and England.
Just did a bit of searching, as I was curious, try this URL:
[uneca.org...]
The current international bandwidth stands at 90Mbps, of which 60Mbps are connected directly to the United Kingdom and United States.Most of the links are carried through the SAT-2 fibre cable across the Atlantic to the US
So, I'd give some serious consideration to hosting in the eastern U.S.
(Note that the data on that site is probably out of date. I saw a reference to "by 1999...")
Just did some traceroutes to a S. Africa Internet provider. Have to say those are some long ping times! (350mSec) I see that in this particular case, though it went N.Y -> London -> S. Africa. London->S. Africa is about 150mSec.
-------
OK, just did some poking around on said Internet provider's web site.
It appears to me that dater-center hosting itself isn't all that much more in S. Africa than in the U.S. For example, I see shared hosting with all the bells and whistles (PHP, MySQL, etc.) for 100R (about $30)/month.
You do pay .09/meg for traffic on top of that. Depending on your traffic - ouch!
In the U.S. many $10/month web hosts tout "1000GB traffic!". Of course, that's unrealistic, and you probably won't get good service from sombody who makes that claim. But 100GB is reasonable. Or let's just say 10. At .09/MB, that would come to $900/month!
I see some companies marketing to S. Africa touting "reliable UK servers". Based on my traceroute experiments, that might be your best way to go. UK-based hosting is I understand pretty competitive, at least approaching U.S. prices.
"Shared hosting with all the bells and whistles" = 200R, not 100R. But still about $30USD ($27.15).
Traffic is, of course, .09R/MB, not .09USD/MB. So, 10GB traffic works out to about $122USD/mo.
Still, that's $150/mo for what you can get for $10/month is the U.S. or England.
The reason I used 10GB traffic is that the "traffic" figures touted by hosts offering "1000GB traffic" are largely fictional. They aren't based on actual traffic measurement, but on a limited bandwidth. They represent the amount of traffic that MIGHT be passed if you utilized the bandwidth 100% of the time, 24 hours/day, 7 days/week.
It appears in S. Africa the standard is to charge for actual traffic passed.
Just so you know, I am the owner of the second largest Widget website <snip> in South Africa (there are about 6 others). Second largest but fastest growing! My website is 8 months old and doubled in size last month. This months growth is already at over 60%. For my country, that's excellent growth! I'm now at over 10,000 visitors and over 100,000 page views per month (bearing in mind that I grow at 60% a month).
The content in the article you sent has changed drastically since 1999! Here is some casual info so you understand the situation in my country from a bandwidth and provider perspective! The largest and ONLY fixed line provider in our contry is called Telkom. They have a monopoly! All ADSL connections are through them (although there is another option for internation bandwith which is Internet Solutions via satellite to the US, this is not always ideal and they charge more)!
We now have 4 cell phone operators:
1) Vodacom (the largest and partly owned by Telkom and has ties to Vodaphone)
2) MTN
3) Cell-C (low budget)
4) Virgin Mobile (recently joined the scene a few months ago).
Because there is more competition in the mobile scene, mobile/wireless internet options are very competitive (but they still don't hold a candle to ADSL). Our country was the first in the world to implement HSDPA at 1,8MB, at that time, the mobile service providers could advertise "the fastest internet connectivity in South Africa"!
We also have a few specific wireless internet providers:
There are ONLY 3 ADSL options in our country!
1) 384 downstream, 128 upstream
2) 512 downstream, 256 upstream
3a) 1MB downstream, 256 upstream
However, the last package has been upgraded in the last 3 months to:
3b) 4MB downstream, 384 upstream
The reason for this ADSL upgrade from 1MB to 4MB was mainly because of HSDPA and the mobile operators advertising the "fastest" internet, because HSDPA at 1,8MB was "faster" than Telkoms 1MB ADSL offering. We have NO SDSL, VSDL, T1's or cable!
The options for internet at home are:
1) I-burst wireless - 1MB
2) Dial-up (still sold with every PC and VERY common)
3) Telkom ADSL (3 options) - becomming VERY popular as it should!
4) 3G or HSDPA at 1,8MB (from one of the mobile operators)
5) ISDN (64k or 2x64k to make 128k)
Anyway, the funny thing is this. MANY websites in our country are still hosted on a 64K Diginet line which could cost you R4000p/m ($570). That's $570 for 64k of bandwidth! Now I know in the states you can probably get a T1 line for less! That's a dedicated line and we don't have anything remotely like a T1 line here! Most people who need fast connectivity in the form of a T1 line, will actually just host their sites as a Dedicated server in the hosts Data Center, and only about 10 hosts have the bandwidth to match a T1. You basically can't get more than a 64k or maybe 128k line to your premesis! The main reason most sites didn't even need more than 64K or 128K is because most people were just using Dial-up! ADSL is fairly new here, like maybe 2 years since they brought out the 192k offering which has now been upgraded to 384k! Currently, funny enough, my host, who is one of the largest in the country, hosts my website on a 512k ADSL line (256K upstream) with a Static IP. My website at home is THE SAME SPEED as it is at the host, and they're one of the biggest! Because ADSL bandwidth is MUCH cheaper than other options, especially if you don't even have a T1 option!
At home, I have the 512k ADSL (256k upstream), my website hosted at home is faster than about 95% of the other hosting companies in our country!
Luckily I'm only about 2 miles from the ADSL exchange I connect to, I live about 20 miles North of Johannesburg. What I am planning on doing is upgrading to the 4MB offering me 384k upstream. Most people in the country who go for ADSL, get the cheapest one which is 384 downstream! So my upstream rate on my 4MB home ADSL will match the downstream of most ADSL clients! When I upgrade, I will already have MORE bandwidth than about 95% of the hosts in South Africa! Compare their 64k/128k Diginet connections to my 4MB/384k home ADSL and you'll see my reasoning! Also, I'm planning on putting DUAL 4MB ADSL lines, with a D-Link Load Balancing router to give me a total of 748 upstream. This type of bandwidth speed will cost about $3000 here if using multiple Diginet lines.
Ok, so in USA you guys enjoy the benefit or T1 and more. Here, ADSL is the fastest bandwidth and connection and used by my hosting company anyway to give my site one of the fastest connections!
The reason I want to do it myself? Yesterday my website went down for 18 hours! THEY DON'T CARE! They've got thousands of other websites! This is the SECOND time this month! Hosting oversees is out of the question! As you can see from the hops and time taken! And also, it is faster from USA to go via UK to SA then directly ... don't ask me why ... but we've got a MUCH better connection to the UK than US (might just be the load on the line to the US)! I just want SPEED at the best price! And I'll get it with dual ADSL lines at home (because NOBODY in the country offers DUAL ADSL with a Static IP)! Also, when my website expands, I just add more ADSL lines for cheap bandwith and load balance them. Then eventually Telkom might upgrade to 8MB/512k which would be GREAT for my website, especially if I get DUAL or multiple lines! Also, I'm planning on getting the second line from another ISP for fault tolerence.
Security: There are multiple firewalls to get to my website! First, both ADSL modems have built in firewalls (not that they're any great), then the load balancing router has a firewall, then I'm running iptables on the actual web-server. IMHO, I can do a better job of hosting, security, setup and config of the web server and traffic than 99% hosts in my country can!
Anyway, I'm hoping someone will also comment on the Search engines (esp. Google), will they still index my content and will there be any major obstacle as far as the IP dropping and changing ... what are the other obstacles?
Here are some links:
<snip>
Thanks again for the info and help jtara!
[edited by: trillianjedi at 11:49 am (utc) on Sep. 20, 2006]
[edit reason] Please read our TOS, thanks. [/edit]
But if you intend to use Google Adwords to draw customers to your site, your plan could indeed cause a problem. If the Adbot visits and your site and a target URL is inaccessible, your ad campaign will be automatically disabled. You then have to manually re-enable it. Not sure if further action is taken if it is repeatedly inaccessable. In any case, the Adwords TOS specifically mentions continuous availability. And the Adbot does periodically re-visit - not just when initially placing an ad.
----
Ok, so in USA you guys enjoy the benefit or T1 and more.
T1? Who uses T1? ;)
In the U.S., most web sites are located in large co-location centers known as "carrier hotels". They are generally run by a neutral party who buys, builds, or leases a large office building (or space in a large office building) with the appropriate facilities (adaquate power, backup power, emergency generators, large air-conditioning capacity, etc.), accessible to large fibre loops.
They then invite major carriers to bring in their lines, leasing them space in the building. Fibre lines and Ethernet cables are run around in the building, terminating in a "meet-me room". They also lease space to various second-tier carriers, telephone companies, cell companies, Internet service providers, web hosting companies, etc. many of whom further sub-lease space.
It's not unusual for such facilities to include access to 30 or more carriers, at speeds to to 1gbit/sec or more.
Given this, you can see why one would have to have pretty compelling and odd reasons to want to host a site in the U.S. on a home DSL line.
Here, ADSL is the fastest bandwidth and connection and used by my hosting company anyway to give my site one of the fastest connections!
I think you may be mistaken on this, and/or perhaps you should seek-out a different hosting company.
I have to assume that the same sort of facilities exist in S. Africa, though certainly not on the same scale as in the U.S. The reason I am so certain of this is that your competitive cell phone system could not exist without these sort of facilities backing them.
Looking at the website of the provider whose prices I quoted yesterday, they say their data centers are directly connected to their own fibre network. Now, whether their services are affordable, or whether they have effective connectivity to other ISPs through JINX (which has limited interconnect bandwidth) is another matter.
-----
There's a complex interaction between latency, bandwidth, burst bandwidth, etc. It all depends on what you are trying to accomplish. I've studied this quite a bit, as I've been involved in developing software for arbitrage stock trading, working with all the major exchanges and ECNs. In that application, latency is everything, and it took some convincing in some cases to wean them from the idea that bandwidth is everything. Now, if you are arbitrage-trading, LATENCY is everything.
With recent changes in browsers and server (simultaneous retrieval of images, speculative retriving of linked pages) you really need burstable bandwidth and at the same time the need for low latency is much less than it was in the past.
So, you see, you have the opposite problem. BURST BANDWIDTH is everything. (You don't have enough traffic to be concerned with overall bandwidth - just burst.) Latency is less important. The problem with a home DSL line is that you don't have the ability to burst to high bandwidth rates.
I do note that the website of SA Rugby comes up just fine here in San Diego, despite the 350mSec ping time. It's a bit pokey when I click on a story, but then the whole page comes up nicely all at once.
I'd suggest that you give the idea of hosting in England a chance. 150mSec is not enough latency to cause users to perceive your site as slow, and is still considered quite acceptable for cross-country performance in the U.S. For example, my pings from San Diego to the NY Times web site run around 150mSec.
There are a number of web sites that will allow you to perform ping and traceroute tests from various parts of the world. Do a search, and you will find them. This will help you in evaluation.
The other, obvious, thing to do is to test various English web sites to see how acceptable their performance is. I'd caution you when testing the websites of web hosting companys to use traceroute to make sure their server is where you THINK it is. The sales website of the hosting company may not be in the same place as those of their customers. Get the names of customers (usually some are available in the host's marketing material) and test the CUSTOMER's websites, again verifying path with traceroute.
In any case, for the reasons stated above (browser behaviour - burst bandwidth is more important than latency) I think you would be better served by either a local, data-center-located web host (if affordable) or an off-shore provider.
Thanks so much for the valuable info, I really DO appreciate it and I hope that once again you can take the time and help!
This is the scenario:
My front page on average is 185KB in total (60KB HTML + images)
A visitor will download 263KB (per visit)
A visitor will view 9 pages
That makes an average of 30KB per page
There are about 100 elements on the front page (images, HTML, CSS etc.)!
The front page will normally be the slowest page to download, because it's the "biggest" in terms of download size, the visitor will then cache about 90% of the images + css after the front page! Zip is enabled!
The problem is, the front page takes 13-15 seconds to load on a home ADSL 384kbps (48 KBps) line at best! 25-35 seconds at peak time!
Theoretically, in PERFECT conditions it should take less than 4 seconds to view the front page! But it's 4 times slower than that! I suspect the site (my host) can only push data at max 128kbps (16 KBps) or they may have throttled the bandwidth (mod_thottle is enabled)
I don't really want to reduce the image count, what I want is a visitor to receive data at his maximum speed!
This is why I still believe 2x 256kbps (upstream) ADSL connections, on 2 fast/good lines with load balancing will outperform my host in my test scenario!
I really don't care much for total bandwidth usage per month! 5GB/10GB doesn't matter! What I want is speed! I want a first time visitor to my website on a 384k ADSL line to feel like he's browsing my website on his LAN! I don't care much for international traffic, since I have very little anyway and the site is designed for local visitors. Also, everyone on ADSL uses the same provider, Telkom. So we're all on the same ADSL network.
This is my theory:
A browser requests a new image from the web server. From the above, I deduce that the average element size is 1.8KB Also, from other sources I have seen an average packet size of just under 1KB. In the ideal scenario, the server sends 2 packets of info for the image over the 2 ADSL lines on 2 separate routes. In the ideal case, the client should receive both packets simultaneously. Around this time the server is sent another request for more info and the packets are again split. And so on ...
I think of dual ADSL like I think of a dual core CPU!
Of course ... nothing is perfect ;)
PS: I feel stupid! Like I'm really missing something here!
So if you want to ping/traceroute it and test an implementation of ADSL in our country ... that's the one! They use an "uncapped" ADSL account because they also have high bandwidth requirements (driver/manual downloads etc.)
For me, wherever I have visited that site (work/home/friends), it's always fast!
I suspect the site (my host) can only push data at max 128kbps (16 KBps) or they may have throttled the bandwidth (mod_thottle is enabled)
That certainly makes sense. If they are hosting websites on a DSL line (crazy, IMO, but that seems to be the way a lot of people do things there) it certainly would be prudent for them to throttle bandwidth to individual sites.
BTW, the D-Link site is almost completely useless here. It took well over a minute to load. But, of course, their target audience is not in the U.S.
This is why I still believe 2x 256kbps (upstream) ADSL connections, on 2 fast/good lines with load balancing will outperform my host in my test scenario!
This is going to be a difficult scheme to implement.
Load-balancing doesn't work quite the way you think. There are a few different kinds of load-balancing, most of which are not applicable to your situation. Most typically, it deals with high server load. A load-balancing server or appliance sends requests to multiple servers behind the server/appliance. Scratch that one off.
Another way is to have your DNS server load-balance. (This is avaialable at many third-party DNS services.) This won't accomplish what you want either. It will just send each user off to a different address, but they will get all their images over the same path.
The only way I see to do it is to changes your URLs so that half of the images come from one address and half from the other. You could do this statically (just edit your files) or dynamically (an Apache module that modifies the URLs on the fly). Dunno if such a module exists. But this may NOT be good for SEO! You may need yet another hack to make sure you serve robots from a single, stable address.
I really don't care much for total bandwidth usage per month! 5GB/10GB doesn't matter! What I want is speed!
Right. Exactly what I am saying. You need a high burst bandwidth - preferably in the several megabit range, not 512K. You need to make sure your host isn't limiting your burst bandwidth with mod_throttle, router throttling, etc.
There ARE hosts in S. Africa that do not use DSL to connect to the net, but are connected directly into the fibre backbone (such as it is). The aforementioned rugby site is hosted at one. (Sticky me if you can't figure it out...)
BTW, I ran some speed tests from San Diego to S. Africa. The best I got was 896Kbit/sec to Gamco. I got 500kbit/sec to Telekom, and 456K to Sentech. You could host here and you'd be doing better than you are now.
But I certainly wouldn't suggest hosting on the W. coast of the U.S. I'd still suggest hosting in England - bandwidth to there is likely to be much higher than to here. Of course, you aren't in a position (on a DSL line) to really test the available international bandwidth. It is almost certainly much, much more than your DSL speed.
Test some English web sites and see if they have acceptable performance from a user's standpoint. That's ultimately what matters. I would urge you to cast away assumptions, poke around, see what "works" and not assume that what "everybody" is doing is the best way.
