Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Is this format ok please ?

9:27 pm on Jan 22, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 26, 2004
posts: 412
votes: 0

Been working on this for a while and feel I'm almost there. If you didn't want any images on your site to be indexed by the search engines and you only wanted Google, Yahoo, MSN and Ask jeeves to visit your website, would the below be ok please ?

User-agent: *
Disallow: /images/
Disallow: /a_images/
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow: /

User-agent: Yahoo-MMCrawler
Disallow: /

User-agent: psbot
Disallow: /

User-agent: *
Disallow: /

User-agent: Googlebot

User-agent: Slurp

User-agent: msnbot

User-agent: Teoma

Any help appreciated.

11:49 am on Jan 25, 2011 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 360

it looks like you have two robots.txt files there. you have "User-agent: *" listed twice. you should only list it once.
11:09 am on Feb 25, 2011 (gmt 0)

New User

5+ Year Member

joined:Jan 20, 2011
votes: 0

Yes i agree with goodroi.You are using two times robots in this format.Please use this for once.I think the rset is fine and ok.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members