homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Is this format ok please ?

10+ Year Member

Msg#: 4256763 posted 9:27 pm on Jan 22, 2011 (gmt 0)

Been working on this for a while and feel I'm almost there. If you didn't want any images on your site to be indexed by the search engines and you only wanted Google, Yahoo, MSN and Ask jeeves to visit your website, would the below be ok please ?

User-agent: *
Disallow: /images/
Disallow: /a_images/
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow: /

User-agent: Yahoo-MMCrawler
Disallow: /

User-agent: psbot
Disallow: /

User-agent: *
Disallow: /

User-agent: Googlebot

User-agent: Slurp

User-agent: msnbot

User-agent: Teoma

Any help appreciated.




WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 4256763 posted 11:49 am on Jan 25, 2011 (gmt 0)

it looks like you have two robots.txt files there. you have "User-agent: *" listed twice. you should only list it once.


Msg#: 4256763 posted 11:09 am on Feb 25, 2011 (gmt 0)

Yes i agree with goodroi.You are using two times robots in this format.Please use this for once.I think the rset is fine and ok.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved