Page is a not externally linkable
blend27 - 2:23 pm on Feb 3, 2013 (gmt 0)
Just caught a crawl attempt from CODERO IPs.
Requests are made from several IPs, 1 second interval.
216.55.161.64
216.55.164.20
206.225.81.153
206.225.81.153
206.225.82.23
216.55.181.182
206.225.94.38
216.55.137.46
216.55.170.28
206.225.93.187
206.225.85.218
206.225.85.162
216.55.181.242
206.225.83.217
216.55.162.116
216.55.161.64
216.55.170.28
With the Same UA: User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20100101 Firefox/14.0.1
No supporting files, no robots.txt.
Now the interesting part, all IPs when visited redirect to the same shopping site(dudool). The crawler excepts and keeps http Cookie, passes it back on the next visit and provides valid site referrers.
But the bot runner still does not know that his crawler spits out headers that are not humanly possible, all though it really looks like a human visit when looking at the headers.