Forum Moderators: open
Have any of you seen this user agent before? It hit the index page on each of my websites. It did not read robots.txt; but I suppose one could argue that would be unnecessary for an index page.
Is this what I think it is? A crawler that somehow checks the Google PageRank for the home pages of websites?
It used a RIPE IP Address: 84.180.99.**.
Thanks for any insight into this crawler.
Note to volatilegx: Please obscure the IP if you feel it's warranted. Thanks. :)
[edited by: volatilegx at 6:17 pm (utc) on Sep. 4, 2005]
[edit reason] obscured IP address [/edit]