Welcome to WebmasterWorld Guest from 54.162.155.183

Forum Moderators: goodroi

Message Too Old, No Replies

Will the following robots.txt disallow all robots BUT Google?

     
10:33 am on Apr 7, 2004 (gmt 0)

10+ Year Member



Just wondering, will the following robots.txt correctly disallow all robots but Google:

User-agent: *
Disallow: /
User-agent: GoogleBot
Disallow:

Logically I think this is correct, though on Google's FAQ, they seem to want us to use "allow" instead of "Disallow:"

Thanks for any insight.

2:11 pm on Apr 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



From the way I understand it, Googlebot needs to see it's rule first. I usually code mine around the following example:

# Keeps Googlebot out of targeted pages & private stuff
#
User-agent: Googlebot
Disallow: /content/
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/
#
# Keeps the rest of the engines out of private stuff
#
User-agent: *
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/

Hope this helps.

1:15 pm on Apr 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yeah, when Googlebot visits your robots.txt file, it needs to know first whether its allowed or not. So if you enter the "dis-allow all robots code" first, it will simple read just that and not whats after it. But if it knows that its allowed, it will read and follow that, forgetting about the next line of code which dis-allows all robots.

Here's the valid code which dis-allows others while allowing only Googlebot:

User-agent: googlebot
Disallow:

User-agent: *
Disallow: /

Alternatively, you can use this one [searchengineworld.com] which will allow only the spiders which are "nice".

Sid

 

Featured Threads

Hot Threads This Week

Hot Threads This Month