Forum Moderators: open

Message Too Old, No Replies

Suspicious crawling

Any ideas what this is?

         

Thez

10:13 am on Jun 10, 2008 (gmt 0)

10+ Year Member



Getting lots of requests from ip 189.61.61.* (*.virtua.com.br) just now, doesn't identify as a crawler bot. Altough it's from the same IP the user-agent string seems to change between each request. No referer, but every 50th request had referer set to 'fillbest.com' which seems to be some online pharmacy site. Needless to say I blocked it right away.

Any ideas what this is ? Some email grabber bot ?

[edited by: incrediBILL at 6:45 pm (utc) on June 11, 2008]
[edit reason] Obscured IPs [/edit]

janharders

10:23 am on Jun 10, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



could be email-grabbing, could be content-stealing, could just be someone testing a bot. we've had quite a few more or less legit bots crawling our sites, coming from universitys. turned out, they were doing a project on some new search engine and thought including robots.txt-support was the last thing to do.
I'd generally set up a bot-trap, aka hidden link to some directory. deny it in robots.txt so google & co won't request ist and hide it via css so visitors won't either. anyone who requests it than is a potential harmful bot and can be banned.