homepage Welcome to WebmasterWorld Guest from 54.227.215.139
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
MSN bot hitting 1000s of pages on my site
New to this forum. Is this just a test for MSN?
jp_css

10+ Year Member



 
Msg#: 62 posted 2:28 am on May 6, 2004 (gmt 0)

MSN bot has been hitting my site on and off until now I have found that it has been indexing through over 1000 pages. Is MSN bot only an experiment for now, or is it something more?

 

Imaster

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 62 posted 6:08 am on May 6, 2004 (gmt 0)

Its been hitting hard on my sites as well. In some cases, even higher than googlebot.

I believe it's an experimental project which they plan to launch few months down the line, which could easily postpone to much more time than that.

Useful Links:
MSN - MSNBot [search.msn.com]
Ballmer: MSN Search - Coming Next Year [webmasterworld.com]
Google Results for 'MSNBOT' on WW [google.com]

uncle_bob

10+ Year Member



 
Msg#: 62 posted 9:09 am on May 6, 2004 (gmt 0)

I've got msnbot deep searching a new site within days of it going live. Googlebot it still just getting the startpage .

pontifex

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 62 posted 12:21 pm on May 6, 2004 (gmt 0)

This thing is much uglier behaving than G!
I just opened a trouble ticket with my provider, because MSNBOT hammered the machine to death.
For the next few hours 65.54.164.* is blocked!
P!

jp_css

10+ Year Member



 
Msg#: 62 posted 2:21 pm on May 7, 2004 (gmt 0)

Yeah, it was really hitting my site very quickly, but slowed down to a crawl now.

MikeBeverley

10+ Year Member



 
Msg#: 62 posted 8:37 pm on May 15, 2004 (gmt 0)

Anyone seen this guy:

msnbot/0.11 (+http://search.msn.com/msnbot.htm)

he has hit my server 28,411 times in the past 14 days. That's alot of server load but I don't want to ban him as I'd like my site included in the Microsoft search engine when it comes out. Anyone know how to slow him down?

mars9820

10+ Year Member



 
Msg#: 62 posted 11:18 pm on May 15, 2004 (gmt 0)

[search.msn.com...]

How often will MSNBot access a page from my web server?

In general MSNBot should not try to access your site more than once every few seconds. MSNBot will also account for the time it takes to download a page from a site so tht if your site has a slower connection we will not access it as frequently. If you find that we are placing too high a load on your site please let us know by sending us e-mail at msnbot@microsoft.com.


erthlng

10+ Year Member



 
Msg#: 62 posted 11:05 pm on May 18, 2004 (gmt 0)

Seeing similar high activity over the past week. About 12,000 pages a day getting crawled.

Harry

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 62 posted 10:55 am on May 19, 2004 (gmt 0)

And it cost me 2.28 gig in one day and used up my allowed bandwidth for the month.

dwilson

10+ Year Member



 
Msg#: 62 posted 11:01 am on May 19, 2004 (gmt 0)

Those having this problem ... are you using dynamically generated pages? with?id=####?

I wonder what similarities there are between sites w/ the difficulty. Perhaps something makes MSNBot think there are more pages than there are. Here's an idea ...

If page.asp?item=1234&last=5432 has a link to page.asp?item=8897&last=1234 etc. ... then a nearly infinite combination of URL's would exist.

Are your sites set up anything like that?

Harry

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 62 posted 11:30 am on May 19, 2004 (gmt 0)

Except the message boards and the shopping cart, everything is plain html

caine

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 62 posted 9:50 am on May 20, 2004 (gmt 0)

MSN is the biggest crawler in my logs at the moment - outdoing everyone together!

helenp

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 62 posted 12:48 pm on May 20, 2004 (gmt 0)

Today seems have stopped spidering that heavily,

jasontromm

10+ Year Member



 
Msg#: 62 posted 7:15 pm on Jun 26, 2004 (gmt 0)

I thought I had banned msnbot from my site, but it's come back.

I put the following lines in my robots.txt file:

User-agent: msnbot*
Disallow: *

And I even banned the IP address it seems to be coming from in my .htaccess file:

deny from 65.54.164.

What else can I do? It's hogging up my bandwidth and I don't care to be in Microsoft's directory. (I hate Microsoft in general) The majority of the world uses Google and people can find trommetter.com that way.

Oliver341

10+ Year Member



 
Msg#: 62 posted 9:46 pm on Jul 19, 2004 (gmt 0)

Old thread, just came across it...

I thought I had banned msnbot from my site, but it's come back.

I put the following lines in my robots.txt file:

User-agent: msnbot*
Disallow: *


Your syntax is all wrong.

Use:

User-agent: msnbot
Disallow: /

Receptional

WebmasterWorld Administrator receptional us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 62 posted 6:48 pm on Jul 21, 2004 (gmt 0)

Anyone know how to slow him down?

Absolutely - MSN built a special robots command "delay" into their msnbot for this purpose - launched in this very forum a couple of weeks ago :)

[webmasterworld.com...]

Dixon.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved