- Crawler, Spider, and User Agent ID forum at WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

BlueSky

4:02 am on Oct 27, 2003 (gmt 0)

This guy didn't ask for robots.txt. He went straight to one of my popular pages. Since I have the word "research" banned for UA's, he got a 403 immediately after which he went away. Not sure his purpose. Google shows this one appearing in a number of stats pages. He uses different IPs. Just skimming thru the results, I saw one from another net in Nigeria and one from Great Britain.

81.199.85.109 - - [26/Oct/2003:18:57:55 -0600] "GET /dir/page.htm HTTP/1.0" 403 294 "-" "Green Research, Inc."

inetnum: 81.199.82.0 - 81.199.87.255
netname: CIDR-COMMUNICATION-01
descr: Internet service provider
country: NG
person: Chikezie Okwunakwe
address: 21 Mobolaji Bank
address: Anthony Way Ikeja
address: Lagos
address: Nigeria

Thanks for those who posted info on Web Content International. He came in pounding today, but got nothing but 403's.

dcrombie

7:24 pm on Oct 30, 2003 (gmt 0)

I'm seeing traffic from them now on 80.179.254.186. It's taking a page every 30 seconds - no images or css.

Looking back in the logs we have hits on seven sites from all over the place, including Nigeria, Panama, Mozambique and Israel...

RewriteCond %{HTTP_USER_AGENT} "^Green Research" [OR]
...

jomaxx

5:55 am on Oct 31, 2003 (gmt 0)

It didn't ring a bell, but I went back and did find "Green Research, Inc." in my logs for this month. But I had already banned both of the IP's you mention recently for frequent (although not heavy) spidering with no regard for the robots.txt.

I'm also seeing what are obviously the same spiders crawling the same list of URL's, coming from IP's that are widely separated geograpically - frequently with names like Satcom and Sky2net. I wonder if these are just proxy servers that have become known to a bunch of email harvesters.

BlueSky

dcrombie

jomaxx

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week