bobmark - 5:11 pm on Sep 30, 2002 (gmt 0)
No, Google is not a charity - that is the whole point.
Google is a business that profits (through adventising and syndication of their results to aol etc.) from the hard work of independent webmasters. WE are the SOLE PROVIDERS of their content. In essence, they are selling our product in the form of search results.
For this we receive "referrals." Google does not pay us in any other form for the use of our content. Think what position Google would be in if every website blocked them from crawling through their robots.txt file (and I do not think this is going to happen nor am I advocating it).
As such they should acknowledge the implicit business relationship between independent websites and themselves and not arbitrarily make changes that affect the incomes of hundreds, if not thousands of us. It is a very unusual situation where content providers are given nothing but the most vague and general guidelines by which to make decisions that affect their economic survival.
I honestly see the day coming when a real, fully supported association of independent websites - and ask yourself, what other industry has no association to lobby for them, interface with monopolies and otherwise protect their interests - comes into being as a result of the arbitrary behaviour of large corporations whose entire business model depends on our content.
You want to regard Google et al as a kind of benovelent society that helps us out, go ahead. But it is NOT the reality.