robots.txt

disllowing googlebot

[dougs]

Hello

We are trying to disallow the bot form parts of our site, but form reading here and observing some sites that have tried the bot doesn't seem to care about this file.

Does anyone have any suggestions.

Cheers

Doug

[Alternative Future]

Do you have access to mod_rewrite?

There are some posts on this here at WW how to disallow and ban bots from parts of your site.

Just a thought!

-George

[ciml]

Googlebot follows Robots Exclusion Protocol very well. URLs excluded in /robots.txt will not be fetched, but they can still show in the SERPs as URLs without titles or snippets.

If you want to prevent URLs from appearing, you're better allowing Google to fetch them in your /robots.txt and then using the robots META tag to remove the page from Google.

Alternatively, you can send the bots away using HTTP as George suggests, but I'd be inclined just to make the links to those URLs non-followable (eg. Javascript or USER_AGENT cloaking.

[dougs]

Cheers CIML.

A genius as ever:)

Doug