Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor
Is this format ok please ? Dexie msg:4256765 9:27 pm on Jan 22, 2011 (gmt 0) Been working on this for a while and feel I'm almost there. If you didn't want any images on your site to be indexed by the search engines and you only wanted Google, Yahoo, MSN and Ask jeeves to visit your website, would the below be ok please ? User-agent: * Disallow: /images/ Disallow: /a_images/ Disallow: /cgi-bin/ User-agent: Googlebot-Image Disallow: / User-agent: Yahoo-MMCrawler Disallow: / User-agent: psbot Disallow: / User-agent: * Disallow: / User-agent: Googlebot Disallow: User-agent: Slurp Disallow: User-agent: msnbot Disallow: User-agent: Teoma Disallow: Any help appreciated. Dexie
goodroi msg:4257792 11:49 am on Jan 25, 2011 (gmt 0)
it looks like you have two robots.txt files there. you have "User-agent: *" listed twice. you should only list it once. kamalsingh msg:4272218 11:09 am on Feb 25, 2011 (gmt 0)
Yes i agree with goodroi.You are using two times robots in this format.Please use this for once.I think the rset is fine and ok.