Saving bandwidth no longer an issue?

[glengara]

At one stage we were being encouraged to use things like if-modified-since to save G bandwidth when crawling, I haven't seen in mentioned in ages, is bandwidth no longer an issue with G?

[Robert Charlton]

...is bandwidth no longer an issue with G?

To bump this up and toss in a thought....

The Supplemental Index seems to be all about bandwidth. I'm assuming that Google has either always used PageRank to throttle its crawling, or that the crawling pattern is implicit in the PageRank algo in some way. Googlebot has crawled "less important" or less prominent pages less often than more prominent pages. The Supplemental Index seems to be formalizing this.

I've often thought that the Supplemental index might be vaguely analogous to Inktomi's old Web Map, but not quite the same. I remember that Inktomi had a Best of the Web database, which was their primary index, and also a full Web Map, that contained a link map of the entire web. I know from talking with an Inktomi engineer at the time that Inktomi was jealous of Google, because Google didn't have to make a profit and therefore had all that computing power. In any event, the tiered structure in Inktomi was definitely a bandwidth thing.

Just a guess... but with the size and growth rate of the web now, it's likely that there are getting to be some strains on computing power at Google, and a tiered structure of some sort is an indication of that, or an indication that Google is looking ahead. The difference is that Supplementals can rank... but they're not updated very often unless they receive an infusion of PageRank.

So, I'd have to say "no" to the bandwidth part of the question. I can't comment on "if-modified" at all. I've never paid attention to it.

For more on the old Inktomi structure...

How Inktomi works...
...although some consider Inktomi doesn't "work" at all
[webmasterworld.com...]

[DamonHD]

Bandwidth very definitely IS an issue, and G still urges use of if-modified-size and GZIP page compression.

In fact, we are just about filling up all the excess capacity worldwide put in during the dot-com boom and so telcos are starting to have to lay more fibre like crazy.

And if your users aren't on broadband, eg they are on dialup or mobile, then you will save them time and money on every pageview that you keep trimmed.

Rgds

Damon

PS. Oh, and as I Web-site owner you will save money and improve performance if you can keep your pages efficient.

[edited by: DamonHD at 8:41 am (utc) on April 15, 2007]