This is a cached page of http://www.example.com/index.html from Blekko's web crawl.
Error: No content
end of quote
Course I deny just about everything.
The question was... what range?
184.108.40.206 - - [01/May/2012:08:26:26 +0100] "GET /robots.txt HTTP/1.1" 200 2642 "-" "Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)"
220.127.116.11 - - [01/May/2012:08:30:44 +0100] "GET /robots.txt HTTP/1.1" 200 2642 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)"
I have ranges:
18.104.22.168 - 22.214.171.124
126.96.36.199 - 188.8.131.52
184.108.40.206 - 220.127.116.11
If you go to the link in the bot's UA blekko give you their crawling ranges. One of the better engines in that respect.
199.187.122.nn does not appear to be assoviated with blekko? Could be a coincidence or someone scraping blekko for links?
18.104.22.168 - - [01/May/2012:08:26:26 +0100]
22.214.171.124 - - [01/May/2012:08:30:44 +0100]
Note the times?
This is approximately 3AM EST, and a very slow time for my websites, thus the log entries were consecutive.
Most North American widget folks are sleeping at that time.
The mid and western Euro's that I allow access to are just beginning their days (these widget folks tend be more active in the late-afternoon and evenings) with an approximate 5-6 hour difference.
Too much of a coincidence for me to dis-associate the two, however everybody knows I'm paranoid ;)
Thanks dstiles. Didn't have the 126.96.36.199 - 188.8.131.52
I had the Silicon Valley Colo as a larger block:
184.108.40.206 - 220.127.116.11
And wilderness I'll keep an eye on 199.187.122.* Thanks
wilderness - my point was that the 199.187.122.nn hit was probably not blekko itself but either someone using blekko as a search source (as is common with google and other SEs) or someone running an automated scrape, which could as easily come at that time as at any other.
Given your sassertion re: access activity times, I would opt for the former: using blekko as a scraper search source.
Either way, I don't see blekko itself being the culprit, although I could be wrong. I'd be interested to see any other evidence of association between blekko and databasebydesign.
|I'd be interested to see any other evidence of association between blekko and databasebydesign. |
Unfortunately and in most instances, and after denying a range I no longer continue accumulating references.
Since denying Blekkobot/ScoutJet it now shows up every single day requesting robots.txt 20 to 30 times.
The good news is, while my pages are still listed there, all the cached copies are now gone.