Forum Moderators: open

Message Too Old, No Replies

Anyone recognize any of these IPs?

New spiders visiting "robots.txt"

         

Friday

8:38 pm on Apr 5, 2002 (gmt 0)

10+ Year Member



I'm getting visits to "robots.txt" from some IP Addresses I don't recognize. Unfortunately, no User-agent or domain name is attached.

Anyone receognize any of these?

TIA

64.210.196.195
66.28.98
151.35.170.118
195.5.3.34
208.254.87.133
209.167.50

digitalghost

8:43 pm on Apr 5, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



208.254.87.133- Gigablast
64.210.196.195 - Girafa

DG

msgraph

8:47 pm on Apr 5, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



64.210.196.195 - Girafa.com
66.28.98 - Cogent Comm(Need more specifics)
151.35.170.118 - Interfree35
195.5.3.34 - Ukraine Telecom
208.254.87.133 - Gigablast
209.167.50 - Can't tell do you have a more specific Class C?

wilderness

12:57 am on Apr 6, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



deny from 209.167.50. LinWalker,SevenTwentyfour Incorporated, 209.167.50.22

SevenTwentyfour crawls your site until they detect a 404 then they send you an automated email informing you of a bad link.

When you reply to the email they send another automated mail which doesn't even address your inquiry and promotes their services of crawling.

wilderness

1:01 am on Apr 6, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



<snip>208.254.87.133 - Gigablast>

These folks are very easy to rid yourself of.

Mail me privately and I'll explain.
There is an htaccess addition which handles this and although I read it publicly, it has been so effective that I don't wish any random crawlers to pick up on it.