homepage Welcome to WebmasterWorld Guest from 54.226.180.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt -- block one user-agent, allow all others
luma




msg:1528531
 1:22 am on May 23, 2002 (gmt 0)

I guess that's a rather simple questions, but I can't seem to find a definite answer. Lets say I want to block one "search engine", e.g., ia_archiver and allow all others, is this how my robots.txt would have to look:

# robots.txt for mydomain.com
User-agent: ia_archiver
Disallow: /
# end

Or will I have to add the follwing at the end of the file:

User-agent: *
Disallow:
# end

If a add this, will "ia_archiver" back off as soon as it reads that it's not welcome, or will it interpret the whole robots.txt and match the "global allow" entry?

PS: Hope I am in the right forum. Couldn't find a more appropriate one in the "Forum Index".

 

Mardi_Gras




msg:1528532
 1:44 am on May 23, 2002 (gmt 0)

User-agent: ia_archiver
Disallow: /

That should be all you need. You can always use a syntax checker - Brett has one at [searchengineworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved