| 12:30 pm on Jul 31, 2004 (gmt 0)|
I think they are preparing a new "feature" ...."Pay ..to not be spider r*ped"...sort of the antithesis of SEO...
Maybe a case for reporting them to the FBI for hacking ;)
| 12:35 pm on Jul 31, 2004 (gmt 0)|
I've read reports that their spider is "very aggressive". People aren't kidding.
My robots.txt validates too, so no excuses there.
| 10:17 pm on Jul 31, 2004 (gmt 0)|
No kiddin, it's already eaten 544 pages from me today and its still going... On one visit!
| 5:57 am on Aug 1, 2004 (gmt 0)|
Just caught it right now coming back for all 3000+
'till they get their act together:
deny from 18.104.22.168
beta test somewhere else!
| 3:22 pm on Aug 4, 2004 (gmt 0)|
Amznvibe and netscan -- we take any potential issues with crawling too quickly or violating robots.txt seriously. If you could tell us what domain this occured on or E-mail us at firstname.lastname@example.org we will work to figure out what is happening.
| 4:23 pm on Aug 22, 2004 (gmt 0)|
Well it slowed for a few days, and then today it grabs 2600 pages in 1 visit. Funny part is that I'm not even listed on MSN.....
| 8:39 am on Aug 23, 2004 (gmt 0)|
See this forum post:
Should sort your problem out.