Blekko/Scoutjet

[keyplyr]

While the Blekkobot and ScoutJet are following the deny in robots.txt and requesting nothing further, something from Blekko *is* crawling and caching pages.

I put up the denies for these two bots of theirs when I read their public statement announcing they would *not* support the nocache tag and that they would continue to post cached version of all web pages in their SERP.

After a month or so I just checked and they are displaying fresh cached versions of my web pages, so something else is getting them. Anyone know what it is and what range it's coming from?

I just added this block:

199.87.248.0 - 199.87.255.255
199.87.248.0/21

[wilderness]

This is a cached page of http://www.example.com/index.html from Blekko's web crawl.

Error: No content
end of quote

Course I deny just about everything.

[keyplyr]

The question was... what range?

[wilderness]

199.87.252.51 - - [01/May/2012:08:26:26 +0100] "GET /robots.txt HTTP/1.1" 200 2642 "-" "Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)"
199.187.122.98 - - [01/May/2012:08:30:44 +0100] "GET /robots.txt HTTP/1.1" 200 2642 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)"

[dstiles]

I have ranges:

38.99.96.0 - 38.99.99.255
64.13.159.0 - 64.13.159.255
199.87.248.0 - 199.87.255.255

If you go to the link in the bot's UA blekko give you their crawling ranges. One of the better engines in that respect.

199.187.122.nn does not appear to be assoviated with blekko? Could be a coincidence or someone scraping blekko for links?

[wilderness]

199.87.252.51 - - [01/May/2012:08:26:26 +0100]
199.187.122.98 - - [01/May/2012:08:30:44 +0100]

Note the times?
This is approximately 3AM EST, and a very slow time for my websites, thus the log entries were consecutive.

Most North American widget folks are sleeping at that time.

The mid and western Euro's that I allow access to are just beginning their days (these widget folks tend be more active in the late-afternoon and evenings) with an approximate 5-6 hour difference.

Too much of a coincidence for me to dis-associate the two, however everybody knows I'm paranoid ;)

[keyplyr]

Thanks dstiles. Didn't have the 38.99.96.0 - 38.99.99.255

I had the Silicon Valley Colo as a larger block:

64.13.128.0 - 64.13.191.255
64.13.128.0/18

And wilderness I'll keep an eye on 199.187.122.* Thanks

[dstiles]

wilderness - my point was that the 199.187.122.nn hit was probably not blekko itself but either someone using blekko as a search source (as is common with google and other SEs) or someone running an automated scrape, which could as easily come at that time as at any other.

Given your sassertion re: access activity times, I would opt for the former: using blekko as a scraper search source.

Either way, I don't see blekko itself being the culprit, although I could be wrong. I'd be interested to see any other evidence of association between blekko and databasebydesign.

[wilderness]

I'd be interested to see any other evidence of association between blekko and databasebydesign.

Unfortunately and in most instances, and after denying a range I no longer continue accumulating references.

[keyplyr]

Since denying Blekkobot/ScoutJet it now shows up every single day requesting robots.txt 20 to 30 times.

The good news is, while my pages are still listed there, all the cached copies are now gone.