Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Is this format ok please ?



9:27 pm on Jan 22, 2011 (gmt 0)

10+ Year Member

Been working on this for a while and feel I'm almost there. If you didn't want any images on your site to be indexed by the search engines and you only wanted Google, Yahoo, MSN and Ask jeeves to visit your website, would the below be ok please ?

User-agent: *
Disallow: /images/
Disallow: /a_images/
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow: /

User-agent: Yahoo-MMCrawler
Disallow: /

User-agent: psbot
Disallow: /

User-agent: *
Disallow: /

User-agent: Googlebot

User-agent: Slurp

User-agent: msnbot

User-agent: Teoma

Any help appreciated.



11:49 am on Jan 25, 2011 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

it looks like you have two robots.txt files there. you have "User-agent: *" listed twice. you should only list it once.


11:09 am on Feb 25, 2011 (gmt 0)

Yes i agree with goodroi.You are using two times robots in this format.Please use this for once.I think the rset is fine and ok.

Featured Threads

Hot Threads This Week

Hot Threads This Month