homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Ongoing ia archiver Abuse From China

 9:14 am on Nov 25, 2012 (gmt 0)

It just asks for my home page at last 100+ times a day and won't take 403 forbidden for an answer.,"ia_archiver","/index.html"

This has been going on for many many months now, usually the same IP, day after day, sometimes multiple times in a minute, then every minute, every few minutes, and then it slows down for a few minutes before it starts up all over again.

At this point I'm beyond mystified, I'm just down right annoyed. I'd save a lot of wasted disk space logging this stupid thing if I just dropped them in the firewall but it's kind of amusing at some level.

Maybe I should try randomly 301 and 302 redirecting just their crawler to all sorts of crazy places and see what happens.

I used to redirect some spammers to and that was fun :)



 10:16 am on Nov 25, 2012 (gmt 0)

So did I until I realized that robots don't have to follow redirects. That kinda took the fun out of it :( Except for one batch of Ukrainians who got so wigged out, it really did cut back on their visits.

Someone hereabouts suggested the alternative of redirecting back to the originating IP. Like Return To Sender. If it's a very stupid robot, it might even work.


 10:41 am on Nov 25, 2012 (gmt 0)

I also though about giving them an infinite honeypot and see how many pages they'd grab that contained nothing but a page full of the repeating word "MORON".

Using gzip, assuming they'll accept it, I could compress a gigabyte(s) of data into a very tiny file that when unzipped could cause issues, especially if they unzip a few of them. I did that to someone before but I only did a 1 megabyte page and it compressed down to a couple of hundred bytes but I'd go much bigger for these guys just because they're really asking for it.


 6:52 pm on Nov 25, 2012 (gmt 0)

lol, "" redirect, never thought of that... I like it.

and I like bills idea of MORON.

I remember back the day of BBS systems being able to push through thousands of backspaces on annoying people and overload peoples modems thus bumping them off.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved