80legs on the crawl

[incrediBILL]

I'm starting to see an escalation of activity hitting the firewall for 80legs.

Anyone else seeing this?

I'm expecting it's just a matter of time before they go stealth to actually crawl my site.

I'm watching...

[tangor]

Me three... So far the creepy crawler has honored my whitelist robots.txt... but (having looked at their website and "what it means") I have little faith they will continue that way.

[caribguy]

About 3 weeks ago, I switched from 403'ing all of that to allowing them to take a peek at robots.txt Surprisingly, it has been well behaved so far.

[tangor]

As I said, has honored the disallow (I disallow everything except a few). I just don't like their biz model and have reservations they will continue to be well-behaved. After all, we have goog as an example. :)

[Pfui]

If this bot's new to you, here's a whole lotta backstory: 80legs [webmasterworld.com...]