dstiles - 11:38 pm on Nov 17, 2009 (gmt 0)

No one seems to have asked WHY some web sites are slow to load.

Early this year I was forced to upgrade my 2-year-old server to a faster one with a bigger pipe. The reason? Higher speed broadband coupled, to a degree, with high-speed accellerator (aka scraper) addons in browsers.

The speed of download has at least quadrupled over the past couple of years (note: this is UK). It now takes only a dozen or so simultaneous visitors to over-load a small virtual server, even one running at 100Mbps. Add a few videos, the odd Mbyte flash page or PDF and speed is soon dragged down.

It's not the fact that there is more traffic, although there is to a certain extent (a high and increasing proportion of it bots from google et al). It's that scraper-browsers pull in a lot of pages and images VERY fast. Hit peak time with several doing this and splatt! An hour later, back to a trickle.

Of course, it's not always browsers at fault. I have education-usage sites that are plagued by so-called "security proxies" that take it upon themselves to scrape every page of a site "just in case" - and then come back again an hour later, regardless of caching directives. I can't block them or my client complains.

I'm not even going to go into "illegal" hits from content thieves and their like - I kill most of those at the door.

How many small-scale business and hobby sites can afford to shop around for higher speed hosting? Indeed, how many would even suspect it is necessary? Most, if told, would say, "Ok, it takes two seconds to show a page. So what? I can't afford to pay twice as much to halve the display time. Especially in this recession!"

Indeed, it doesn't matter that much. Most people are happy with a second or two wait: in my experience that is commonplace anyway. So google gets upset if it takes a few hundred milliseconds: that's THEIR problem not ours. The REAL visitors are quite content. If SEs want faster access then a) hit less frequently and violently; and b) visit when the site isn't busy, like 1am in the morning (and don't let them say that can't be done!).

The sites I host now are small. A year or so ago I moved my highest bandwidth site to its own high-speed server just to get ANY usable bandwidth back on the virtual server. It gave me a respite of about 12 months before I had to upgrade the VS. And charge customers for the extra. Which they are not entirely happy about but put up with it when I explain.

When it comes down to it, for whom do we build web sites? Search engines or customers?

SEs are dictating far too much. They want our content because it makes them money. I don't really think they care over-much if it makes us money - in several cases companies have been bankrupted or closed due to an SE screwing around with its algorithms (one of them was almost one of my clients!) and never an apology seen.

I know my rant will not affect the outcome of this - I doubt very much google will even see it. I just wanted to make the (long-winded) point. Sorry.