Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

MSNbot indexes site every day for months, still not the index

at the new MSN preview site



12:56 am on Nov 11, 2004 (gmt 0)

I went and looked back for at least six month (random checking of .log files) and I see that msnbot snatched dozens if not hundreds of pages a day. Still even a search for mydomain.com returns a no results page. Tried all kinds of combinations and see nothing!

is this "msnbot/0.3 (+http://search.msn.com/msnbot.htm)" the current spider? Nothing weird on the robtos.txt, .htaccess and the code returned for the pages it visited is 200.

any idea? Many other sites in the same server are indexed (not many pages though) and there's no interlinking.


1:29 am on Nov 11, 2004 (gmt 0)

10+ Year Member

Are you searching from the tech preview, or msn itself?

The tech preview, while certianly needing some tweaking, sure looks fresh. Our site has been getting updated in teh tech preveiw serps cache every 12 hours all week!


1:44 am on Nov 11, 2004 (gmt 0)

searching from the techpreview...and I'm surprised since if I was banned I doubt they'd crawl daily and the code is 200 so the pages are there.


1:50 pm on Nov 13, 2004 (gmt 0)

Update: My index page is not in the index
but I have about 33 pages in with the 10/29/04 stamp on. Google has about 6000 pages.


6:15 pm on Nov 13, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I feel ya, walkman.

MSNbot sucked up 17.5 gigabytes from our server in May, before I had to ban it. Nothing in the SERPs except our domain name (so the index is aware of us but have no data). We feel brutalized.

I just un-banned it and submitted the URLs, so we'll see what happens.


5:18 am on Nov 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

the bot fetches 10's of thousands of pages every week but for some reason i don't have more then 6k in the tech preview.


2:48 pm on Nov 15, 2004 (gmt 0)

10+ Year Member

Same here - MSNbot has been very good at crawling my site for months now. The pages come up at the beta site on a site:myhobbysite.com search, but not on any keyword search (and trust me, I got some pretty unique keyword combos).

Featured Threads

Hot Threads This Week

Hot Threads This Month