homepage Welcome to WebmasterWorld Guest from 54.237.125.89
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Is there any way to block everyone except.
for Google, Yahoo etc?
mmiller




msg:3606968
 2:01 am on Mar 21, 2008 (gmt 0)

Hi Folks;

Just curious - is there a way to block all search engines except the most popular?

My thinking is block everyone except google, yahoo, microsoft etc.

 

bubster119




msg:3607152
 9:41 am on Mar 21, 2008 (gmt 0)

User-agent: *
Disallow: /

User-agent: googlebot
Disallow:

User-agent: slurp
Disallow:

User-agent: etc...
Disallow:

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved