Msg#: 4186993 posted 4:16 am on Aug 15, 2010 (gmt 0)
Ken, I've emailed you.
FYI, emailing email@example.com doesn't work - I get a warning saying that connection is refused by lb.wordpress.com.
edit: brain fart, obviously emailing ken@<samedomain> isn't going to work either...
edit 2: checked my logs and there is no record of the bixolabs crawler ever fetching robots.txt. The repeated fetching of the same URL ended abruptly around 5 hours ago; at that point it was requesting the [same] URL every few seconds.
Msg#: 4186993 posted 5:04 am on Aug 15, 2010 (gmt 0)
1. Sorry about the problem with email - it's an issue with some email systems doing an MX lookup to resolve the destination domain. Please try firstname.lastname@example.org. And I'll update the user agent string to specify the full domain, to avoid this in the future.
2. I'm fairly certain robots.txt would have been fetched, but it would have been very early in the (15 hour) crawl cycle. What's the domain, and the URL being repeatedly fetched? Does it get decorated by different query parameters?