homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum (MSNBot) sucks down 3000 pages
violates my robots.txt completely

 12:18 pm on Jul 31, 2004 (gmt 0)

Ugh. Apparently MSNbot feels like it doesn't have to obey robots.txt?

I'll have to research more on this but anyone else go through this experience?



 12:30 pm on Jul 31, 2004 (gmt 0)

I think they are preparing a new "feature" ...."Pay ..to not be spider r*ped"...sort of the antithesis of SEO...

Maybe a case for reporting them to the FBI for hacking ;)


 12:35 pm on Jul 31, 2004 (gmt 0)

I've read reports that their spider is "very aggressive". People aren't kidding.

My robots.txt validates too, so no excuses there.


 10:17 pm on Jul 31, 2004 (gmt 0)

No kiddin, it's already eaten 544 pages from me today and its still going... On one visit!


 5:57 am on Aug 1, 2004 (gmt 0)

Just caught it right now coming back for all 3000+

'till they get their act together:
deny from
beta test somewhere else!


 3:22 pm on Aug 4, 2004 (gmt 0)

Amznvibe and netscan -- we take any potential issues with crawling too quickly or violating robots.txt seriously. If you could tell us what domain this occured on or E-mail us at msnbot@microsoft.com we will work to figure out what is happening.


-msndude (msd)


 4:23 pm on Aug 22, 2004 (gmt 0)

Well it slowed for a few days, and then today it grabs 2600 pages in 1 visit. Funny part is that I'm not even listed on MSN.....


 8:39 am on Aug 23, 2004 (gmt 0)

See this forum post:


Should sort your problem out.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved