Welcome to WebmasterWorld Guest from 54.158.65.139

Forum Moderators: mack

65.54.188.108 (MSNBot) sucks down 3000 pages

violates my robots.txt completely

   
12:18 pm on Jul 31, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Ugh. Apparently MSNbot feels like it doesn't have to obey robots.txt?

I'll have to research more on this but anyone else go through this experience?

12:30 pm on Jul 31, 2004 (gmt 0)

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I think they are preparing a new "feature" ...."Pay ..to not be spider r*ped"...sort of the antithesis of SEO...

Maybe a case for reporting them to the FBI for hacking ;)

12:35 pm on Jul 31, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I've read reports that their spider is "very aggressive". People aren't kidding.

My robots.txt validates too, so no excuses there.

10:17 pm on Jul 31, 2004 (gmt 0)

10+ Year Member



No kiddin, it's already eaten 544 pages from me today and its still going... On one visit!
5:57 am on Aug 1, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Just caught it right now coming back for all 3000+

'till they get their act together:
deny from 65.54.188.108
beta test somewhere else!

3:22 pm on Aug 4, 2004 (gmt 0)

10+ Year Member



Amznvibe and netscan -- we take any potential issues with crawling too quickly or violating robots.txt seriously. If you could tell us what domain this occured on or E-mail us at msnbot@microsoft.com we will work to figure out what is happening.

Thanks.

-msndude (msd)

4:23 pm on Aug 22, 2004 (gmt 0)

10+ Year Member



Well it slowed for a few days, and then today it grabs 2600 pages in 1 visit. Funny part is that I'm not even listed on MSN.....
8:39 am on Aug 23, 2004 (gmt 0)



See this forum post:

[webmasterworld.com...]

Should sort your problem out.

Dixon.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month