incrediBILL - 11:33 pm on Feb 3, 2011 (gmt 0)
SiteSpeedBot claims that name is their user agent yet that would never pass my filters.
Yet somehow my data, that shouldn't be there if everything they said is true, magically appears on their servers.
Without wasting any time I decided to see what's up and asked the site to crawl a bogus page.
It didn't ask for robots.txt like it claims, asked for the home page as SiteSpeedBot, then proceeded to ask for the bogus page as Firefox.
220.127.116.11 - "GET / HTTP/1.1" 200 2297 "-" "SiteSpeedBot"
18.104.22.168 - HEAD /bogus.html HTTP/1.1" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:22.214.171.124) Gecko/2009060308 Linux Mint/7 (Gloria) Firefox/3.0.11"
Then I tried again with a page it could find, and it sort of asked for robots.txt, not very clever really...
126.96.36.199 - "GET /page.htmlrobots.txt HTTP/1.1" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:188.8.131.52) Gecko/2009060308 Linux Mint/7 (Gloria) Firefox/3.0.11"
It actually asked for the page HEAD to verify it existed BEFORE asking for the robots.txt file, huh?
What a mess.
However, they claim to operate from multiple data centers around the world, so I'm just scratching the surface of their IPs it would seem.
Here's another IP and UA they used...
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:184.108.40.206) Gecko/20100908 CentOS/3.6-2.el5.centos Firefox/3.6.9 GTB7.1"
It seems they like 1&1 for hosting this thing from different data centers.
Anyone got anything else on this beast?