Welcome to WebmasterWorld Guest from 54.227.5.198

Forum Moderators: goodroi

Message Too Old, No Replies

google allow - disallow all others

     

jimmy19

4:22 pm on Jul 19, 2004 (gmt 0)

10+ Year Member



Does anyone have a sample of how I would set up a Robots.txt file to allow googlebot but disallow all other spiders from the whole web site?

Thanks for the help!

:)

wkitty42

2:07 pm on Jul 29, 2004 (gmt 0)

10+ Year Member



surely you've found the answer to this by now? ;)

sem4u

2:11 pm on Jul 29, 2004 (gmt 0)

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member



You can only disallow spiders if you know the names of them and they will all have to be entered into the robots.txt file.

There is a good list here though:

[webmasterworld.com...]

;)

encyclo

2:41 pm on Jul 29, 2004 (gmt 0)

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Welcome to WebmasterWorld, jimmy19.

The safest way would be to cloak robots.txt to deliver a disallow all to anything other than Googlebot.

The basic process is to use mod_rewrite to redirect calls for robots.txt to, say, robots.php, and in the latter file, check the IP address and user agent string to identify Googlebot, and then print the appropriate robots.txt declarations. You could even place all IPs other than Googlebot accessing the robots.txt file on a banned list to ensure that they can't spider the site.

Rather complicated, but the only sure way I know of!

jimbeetle

3:01 pm on Jul 29, 2004 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



For "well-behaved" robots, those that obey robots.txt, this is the syntax recommended by robotstxt.org:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

jimmy19

7:20 am on Jul 30, 2004 (gmt 0)

10+ Year Member



I will take a look at these. Thank you for the replies...

Jim :)

 

Featured Threads

Hot Threads This Week

Hot Threads This Month