lucy24 - 7:39 pm on Jun 28, 2011 (gmt 0)
Obvious question: has the site in fact changed since 2006?
Rumor has it that some robots get anxious if they can't find a robots.txt at all, and go on to assume the worst. ("I'm not allowed in here.") It may be safer to put a robots.txt in place, even if all it says is
That is, let everyone crawl everywhere-- but you will soon decide you didn't really mean this!
Is the site getting human visitors? What's in the htaccess? It's hard not to think that back around 2006 you absent-mindedly locked them out. This is easy to do, because the ia_archiver is squarely in the middle of the amazonaws block, which a lot of people lock out on principle.
My current .htaccess says, complicatedly,
Fortunately my father taught me base 2 in early childhood ;)
This is still not precise: I'm allowing the whole 129 range except 117 which contains a robot I don't care for.* But so far I haven't met anyone but the ia_archiver elsewhere in 129.
* Nothing wrong with the robot itself, I just don't like their bosses.