IBM_Planetwide

Getting hit very hard by this spider

[requiem]

IBM_Planetwide has been hitting on my my client sites for over 300mb this month is it not a little bit strange concidering the normal behaviour of this spider?

[coyote]

This thing has been around for years and - to the best of my knowledge - isn't related to a search engine. I think it's a crawler IBM uses for experiments (better someone else's bandwidth than theirs) and/or to rip off websites.

300MB is NOT normal SE spider behavior. I'd advise banning by both UA and IP.
Search the forums here and you're bound to find some interesting posts about it.

coyote

[requiem]

Thank you for the answer. I will ban the spider in the next update of my clients .htaccess file.

[moltar]

I had this too.

If you use awstats, then it's a glitch. awstats identifies this spider by 'ibm' string in UA. By looking through my logs I found that many users had ibm in their UA and it was not a spider. I simply deleted that record from UA script.