oddsod - 7:59 pm on Feb 13, 2007 (gmt 0)

After all I think both are stupid. We all work as hard as we can to be in every SE, but they want to be out? Old media.

Not everyone needs Google. Many big sites couldn't care if they never get another Google referral, ever. I used to be where you are but now have sites that really don't need Google so I can now see the Belgians' view.

Over the last few years I took a leaf out of Brett's book and blocked Google from caching my pages. Then I went further and actually blocked Google from some of my content (via robots.txt). Guess what? They didn't give a sh*t and kept crawling that content. So, what do I have to do to block them crawling? Why not give my robots.txt the full respect it deserves and not even venture into folders I've blocked? Talk to me about robots.txt when
a) there are some decent changes made to the standards
b) when Google learns to respect it

Would you like it if every spammer in the world used the excuse that they could spam you because you didn't opt out? It's the same principle. Don't assume I've opted in just because I haven't specifically told you (whoever "you" are) that I don't want to opt in.

It's essentially free advertising.

It's essentially free content.

And so what if they cache it? The net benefit to me is still more traffic.

What makes you think I want traffic? Or, specifically, your traffic. Or your traffic on your terms (deep linked to bypass my homepage ads)?