homepage Welcome to WebmasterWorld Guest from 107.22.70.215
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Will the following robots.txt disallow all robots BUT Google?
georgec




msg:1529001
 10:33 am on Apr 7, 2004 (gmt 0)

Just wondering, will the following robots.txt correctly disallow all robots but Google:

User-agent: *
Disallow: /
User-agent: GoogleBot
Disallow:

Logically I think this is correct, though on Google's FAQ, they seem to want us to use "allow" instead of "Disallow:"

Thanks for any insight.

 

SEOMike




msg:1529002
 2:11 pm on Apr 8, 2004 (gmt 0)

From the way I understand it, Googlebot needs to see it's rule first. I usually code mine around the following example:

# Keeps Googlebot out of targeted pages & private stuff
#
User-agent: Googlebot
Disallow: /content/
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/
#
# Keeps the rest of the engines out of private stuff
#
User-agent: *
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/

Hope this helps.

sidyadav




msg:1529003
 1:15 pm on Apr 9, 2004 (gmt 0)

Yeah, when Googlebot visits your robots.txt file, it needs to know first whether its allowed or not. So if you enter the "dis-allow all robots code" first, it will simple read just that and not whats after it. But if it knows that its allowed, it will read and follow that, forgetting about the next line of code which dis-allows all robots.

Here's the valid code which dis-allows others while allowing only Googlebot:
User-agent: googlebot
Disallow:

User-agent: *
Disallow: /

Alternatively, you can use this one [searchengineworld.com] which will allow only the spiders which are "nice".

Sid

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved