jdMorgan - 11:44 pm on Jun 30, 2003 (gmt 0)

In response to this post, several members have told me that they modified their robots.txt files. I now have one reliable report that ia_archiver is not in compliance with the Standard for Robots Exclusion, and won't accept multiple user-agents in a single shared-policy record. For those who wish to exclude ia_archiver, it must have its own record, e.g.

User-agent: ia_archiver
Disallow: /user_files/

(each record must be followed by a blank line.)

I'd like to hear reports of any more non-compliant user-agents this technique turns up.

Jim