IP address: 38.125.95.250 Reverse DNS: webcrawler.hailoo.com
keyplyr
11:52 am on Aug 18, 2010 (gmt 0)
Note to dev: bot should request robots.txt *before* other files, not hours after.
Dijkgraaf
2:06 am on Aug 19, 2010 (gmt 0)
In my case it did get for robots.txt just seconds before it fetched the page on 4 occasions.
However on all for occasions it was the same page it was requesting, twice on the 16th and once on the 17th and 18th. That's a bit too frequent really.
