DBLBot

dontbuylists.com

[dstiles]

First time I've noticed it but it may have sneaked through before.

Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)

Swept up every page of a single site (about ten pages) at a reasonable rate BUT did not obey robots.txt.

Came from Ukraine range 91.193.166.*

[GaryK]

As a lot of the old timers know I permit anything at all to crawl one of my sites because I want to see what they do. This past week on that site DBLBot did exactly what it did on your site.

[dstiles]

Today it went berserk across three sites, sucking up all or a large percentage of each site despite being fed a 403 on every page.

This time it came from 195.128.18.* - again Ukraine, probably DSL if the Whois doesn't lie.

I've been seeing an increasing number of scrape attempts from Ukraine in the past few weeks.

[phranque]

i noticed dblbot in the logs, so i have added a disallow for it in robots.txt but haven't seen it back since.
it did at least request a HEAD and then a GET of robots.txt priot to any past crawls.

same ip...

[System]

The following 8 messages were cut out to new thread by incredibill. New thread at: search_engine_spiders/3805344.htm [webmasterworld.com]
10:31 am on Dec. 11, 2008 (PST -8)

[Megaclinium]

I had this one hit me too a a few days ago from the 195.128.18.xx IP.
4 pages or so a second, way to quick. I finally added the crawl delay by the way, for those bots that actually look at robots.

No embedded media links grabbed, just text but from the rate they scrape, look like a burglar knowing hte alarm is going off and the cops will be there in 15 minutes

UA same:
"Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)"

Kind of funny, another one time scraper hit me a while back and the lookup said the IP address was a public kiosk in moscow park! Guess they don't lock down those PCs very carefully.