Welcome to WebmasterWorld Guest from 54.144.80.75

Forum Moderators: mack

Message Too Old, No Replies

65.54.188.108 (MSNBot) sucks down 3000 pages

violates my robots.txt completely

     
12:18 pm on Jul 31, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 16, 2002
posts:2010
votes: 0


Ugh. Apparently MSNbot feels like it doesn't have to obey robots.txt?

I'll have to research more on this but anyone else go through this experience?

12:30 pm on July 31, 2004 (gmt 0)

Senior Member from FR 

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Feb 15, 2004
posts:6717
votes: 230


I think they are preparing a new "feature" ...."Pay ..to not be spider r*ped"...sort of the antithesis of SEO...

Maybe a case for reporting them to the FBI for hacking ;)

12:35 pm on July 31, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 16, 2002
posts:2010
votes: 0


I've read reports that their spider is "very aggressive". People aren't kidding.

My robots.txt validates too, so no excuses there.

10:17 pm on July 31, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:May 28, 2004
posts:128
votes: 0


No kiddin, it's already eaten 544 pages from me today and its still going... On one visit!
5:57 am on Aug 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 16, 2002
posts:2010
votes: 0


Just caught it right now coming back for all 3000+

'till they get their act together:
deny from 65.54.188.108
beta test somewhere else!

3:22 pm on Aug 4, 2004 (gmt 0)

Full Member

10+ Year Member

joined:May 23, 2004
posts:261
votes: 0


Amznvibe and netscan -- we take any potential issues with crawling too quickly or violating robots.txt seriously. If you could tell us what domain this occured on or E-mail us at msnbot@microsoft.com we will work to figure out what is happening.

Thanks.

-msndude (msd)

4:23 pm on Aug 22, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:May 28, 2004
posts:128
votes: 0


Well it slowed for a few days, and then today it grabs 2600 pages in 1 visit. Funny part is that I'm not even listed on MSN.....
8:39 am on Aug 23, 2004 (gmt 0)

Senior Member

joined:Mar 8, 2002
posts:2897
votes: 0


See this forum post:

[webmasterworld.com...]

Should sort your problem out.

Dixon.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members