homepage Welcome to WebmasterWorld Guest from 54.226.173.169
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
65.54.188.108 (MSNBot) sucks down 3000 pages
violates my robots.txt completely
amznVibe




msg:1540817
 12:18 pm on Jul 31, 2004 (gmt 0)

Ugh. Apparently MSNbot feels like it doesn't have to obey robots.txt?

I'll have to research more on this but anyone else go through this experience?

 

Leosghost




msg:1540818
 12:30 pm on Jul 31, 2004 (gmt 0)

I think they are preparing a new "feature" ...."Pay ..to not be spider r*ped"...sort of the antithesis of SEO...

Maybe a case for reporting them to the FBI for hacking ;)

amznVibe




msg:1540819
 12:35 pm on Jul 31, 2004 (gmt 0)

I've read reports that their spider is "very aggressive". People aren't kidding.

My robots.txt validates too, so no excuses there.

netscan




msg:1540820
 10:17 pm on Jul 31, 2004 (gmt 0)

No kiddin, it's already eaten 544 pages from me today and its still going... On one visit!

amznVibe




msg:1540821
 5:57 am on Aug 1, 2004 (gmt 0)

Just caught it right now coming back for all 3000+

'till they get their act together:
deny from 65.54.188.108
beta test somewhere else!

msndude




msg:1540822
 3:22 pm on Aug 4, 2004 (gmt 0)

Amznvibe and netscan -- we take any potential issues with crawling too quickly or violating robots.txt seriously. If you could tell us what domain this occured on or E-mail us at msnbot@microsoft.com we will work to figure out what is happening.

Thanks.

-msndude (msd)

netscan




msg:1540823
 4:23 pm on Aug 22, 2004 (gmt 0)

Well it slowed for a few days, and then today it grabs 2600 pages in 1 visit. Funny part is that I'm not even listed on MSN.....

Receptional




msg:1540824
 8:39 am on Aug 23, 2004 (gmt 0)

See this forum post:

[webmasterworld.com...]

Should sort your problem out.

Dixon.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved