While checking through my logfiles this morning I see that the Mirago Robot has taken a large amount of pages (80%) across several sites.
Anyone else notice them crawling?
Craig
mat
9:15 am on Oct 21, 2004 (gmt 0)
Yes, on and off for the last couple of weeks. Yesterday in particular. Not, however, UK sites.
IanTurner
9:19 am on Oct 21, 2004 (gmt 0)
They seem to have been going hard at it just recently.
lots of sites being heavily crawled
davec
1:55 pm on Oct 24, 2004 (gmt 0)
Yep they seem to be taking a lot more pages than usual from our UK sites.
d
ritch_b
11:19 am on Nov 24, 2004 (gmt 0)
Spidering very, very heavily recently - at least, compared to previous activity. :)
R.
ncw164x
6:28 pm on Nov 24, 2004 (gmt 0)
I have always had heavy spidering from Henry but the amount of referals I get in return are not worth talking about, shame really because the staff seem a great bunch of people.
