ia_archiver

How do I get rid of this thing....

[woop01]

I've tried reading about a dozen posts from the search on this spider but for some reason it's not quite clicking with me. I've already banned it via robots.txt but it hit me for 1,200 pages yesterday and from what I understand it doesn't leave you alone until you explicity ban it.

How do I go about banning the ia_archiver bot from an IIS server with 98% ASP pages other than the standard robots.txt file?

Thanks,

[sun818]

Hi, you can look here for an example of ia_archiver:

[webmasterworld.com...]

I did notice there are at least two versions of ia_archiver.

Another possibility is that a rogue bot is "spoofing" its name as ia_archiver and not revealing its true name.

[Dreamquick]

Personally I've banned it via robots.txt and it's learnt to stay out.

In ASP you could put something like this into your pages and have it run before anything else is written;

If Request.ServerVariables("HTTP_USER_AGENT") = "ia_archiver" Then

Response.Status = "403 Denied"
Response.Write "403 - Access Denied"
Response.End

End If

This will block their crawler via it's user-agent information and waste the least possible amount of your server's bandwidth.

- Tony

[wilderness]

<snip>from an IIS server with</snip>

[webmasterworld.com...]

[jdMorgan]

I'm using the following lines (plus more) in my robots.txt, and have had no trouble at all from this 'bot.

User-agent: ia_archiver
Disallow: /private/
Disallow: /logs/

I'm leaning toward the "spoofing" theory posted by sun818... Has anyone else had a problem with the Internet Archive on a site with a validated robots.txt - and verified that the IP address matches the IA range?

Jim

[wilderness]

I confer with Jim here on opposite sides of the fence.
I have ia on disallow in my robots and they do honor that amd remain compliant.

Is it possible woop01' visitor was that iea referrer?

Don

[sun818]

You know, another way to see if this is a rogue bot is to go to [archive.org...] and try to view your site using the service. My understanding is that if you have a current robots.txt that disallows ia_archiver the user will not be able to previous versions of your site even if it was archived.

If your robots.txt is valid and the services prevents you from viewing previous versions, it is a rogue bot.