Welcome to WebmasterWorld Guest from 54.196.238.210

Forum Moderators: goodroi

Message Too Old, No Replies

Is this format ok please ?

     

Dexie

9:27 pm on Jan 22, 2011 (gmt 0)

10+ Year Member



Been working on this for a while and feel I'm almost there. If you didn't want any images on your site to be indexed by the search engines and you only wanted Google, Yahoo, MSN and Ask jeeves to visit your website, would the below be ok please ?

User-agent: *
Disallow: /images/
Disallow: /a_images/
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow: /

User-agent: Yahoo-MMCrawler
Disallow: /

User-agent: psbot
Disallow: /



User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

User-agent: Slurp
Disallow:

User-agent: msnbot
Disallow:

User-agent: Teoma
Disallow:

Any help appreciated.

Dexie

goodroi

11:49 am on Jan 25, 2011 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



it looks like you have two robots.txt files there. you have "User-agent: *" listed twice. you should only list it once.

kamalsingh

11:09 am on Feb 25, 2011 (gmt 0)



Yes i agree with goodroi.You are using two times robots in this format.Please use this for once.I think the rset is fine and ok.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month