google allow - disallow all others

[jimmy19]

Does anyone have a sample of how I would set up a Robots.txt file to allow googlebot but disallow all other spiders from the whole web site?

Thanks for the help!

:)

[wkitty42]

surely you've found the answer to this by now? ;)

[sem4u]

You can only disallow spiders if you know the names of them and they will all have to be entered into the robots.txt file.

There is a good list here though:

[webmasterworld.com...]

;)

[encyclo]

Welcome to WebmasterWorld, jimmy19.

The safest way would be to cloak robots.txt to deliver a disallow all to anything other than Googlebot.

The basic process is to use mod_rewrite to redirect calls for robots.txt to, say, robots.php, and in the latter file, check the IP address and user agent string to identify Googlebot, and then print the appropriate robots.txt declarations. You could even place all IPs other than Googlebot accessing the robots.txt file on a banned list to ensure that they can't spider the site.

Rather complicated, but the only sure way I know of!

[jimbeetle]

For "well-behaved" robots, those that obey robots.txt, this is the syntax recommended by robotstxt.org:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

[jimmy19]

I will take a look at these. Thank you for the replies...

Jim :)