65.54.188.108 (MSNBot) sucks down 3000 pages

violates my robots.txt completely

[amznVibe]

Ugh. Apparently MSNbot feels like it doesn't have to obey robots.txt?

I'll have to research more on this but anyone else go through this experience?

[Leosghost]

I think they are preparing a new "feature" ...."Pay ..to not be spider r*ped"...sort of the antithesis of SEO...

Maybe a case for reporting them to the FBI for hacking ;)

[amznVibe]

I've read reports that their spider is "very aggressive". People aren't kidding.

My robots.txt validates too, so no excuses there.

[netscan]

No kiddin, it's already eaten 544 pages from me today and its still going... On one visit!

[amznVibe]

Just caught it right now coming back for all 3000+

'till they get their act together:
deny from 65.54.188.108
beta test somewhere else!

[msndude]

Amznvibe and netscan -- we take any potential issues with crawling too quickly or violating robots.txt seriously. If you could tell us what domain this occured on or E-mail us at [e-mail] we will work to figure out what is happening.

Thanks.

-msndude (msd)

[netscan]

Well it slowed for a few days, and then today it grabs 2600 pages in 1 visit. Funny part is that I'm not even listed on MSN.....

[Receptional]

See this forum post:

[webmasterworld.com...]

Should sort your problem out.

Dixon.