Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- EC2LinkFinder

iamzippy - 6:22 pm on Mar 11, 2012 (gmt 0)


Hmmm. Another one came by. Different IP, but I had by now updated robots.

107.22.nnn.nnn - - [11/Mar/2012:02:38:19 +0100] "GET /robots.txt HTTP/1.1" 301 251 ..."" "EC2LinkFinder" "-"
107.22.nnn.nnn - [11/Mar/2012:02:38:19 +0100] "GET /robots.txt HTTP/1.1" 200 2353 www.... "" "EC2LinkFinder" "-"

That's all she wrote. So it's clear that it follows canonical 301s and it obeys robots.txt.

Just as well. My EC2 policy is er... robust.

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4427499.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com