Forum Moderators: open
I wonder what my attitude should be considering they didn't read robots.txt and tried to crawl non-existent files? Hmmm... They'll be getting an e-mail with bad attitude from me and then I'll force them to stop crawling my sites, :).
The problem is that *anyone* can potentially use it for *anything*, including crawlers, and that rDNS is generally not provided. I've seen what appear to be legitimate robots from this IP range, so they must be renting additional performance from Amazon for crawling. Unfortunately, since there is no rDNS, I give them the boot. I hope that they'll come back later with an IP range that resolves, but if I drop out of their indexes because I blocked them, then this becomes an 'unsolvable' problem: Too many junk requests to allow non-rDNS-validated robot access, and this hosting model apparently does not support rDNS... :(
Jim
I've seen what appear to be legitimate robots from this IP range, so they must be renting additional performance from Amazon for crawling.
I wish Amazon cared about what things like this do to their good name more than they seem to care only about the bottom line.
Oh well, I've got more important things to worry about right now like today's financial bloodbath.
I'll let them crawl depending on *MY* attitude! :)
[edited by: incrediBILL at 2:32 am (utc) on Oct. 1, 2008]
