homepage Welcome to WebmasterWorld Guest from 54.161.192.61
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
MSNbot indexes site every day for months, still not the index
at the new MSN preview site
walkman



 
Msg#: 222 posted 12:56 am on Nov 11, 2004 (gmt 0)

I went and looked back for at least six month (random checking of .log files) and I see that msnbot snatched dozens if not hundreds of pages a day. Still even a search for mydomain.com returns a no results page. Tried all kinds of combinations and see nothing!

is this "msnbot/0.3 (+http://search.msn.com/msnbot.htm)" the current spider? Nothing weird on the robtos.txt, .htaccess and the code returned for the pages it visited is 200.

any idea? Many other sites in the same server are indexed (not many pages though) and there's no interlinking.

 

WebFusion

10+ Year Member



 
Msg#: 222 posted 1:29 am on Nov 11, 2004 (gmt 0)

Are you searching from the tech preview, or msn itself?

The tech preview, while certianly needing some tweaking, sure looks fresh. Our site has been getting updated in teh tech preveiw serps cache every 12 hours all week!

walkman



 
Msg#: 222 posted 1:44 am on Nov 11, 2004 (gmt 0)

searching from the techpreview...and I'm surprised since if I was banned I doubt they'd crawl daily and the code is 200 so the pages are there.

walkman



 
Msg#: 222 posted 1:50 pm on Nov 13, 2004 (gmt 0)

Update: My index page is not in the index
but I have about 33 pages in with the 10/29/04 stamp on. Google has about 6000 pages.

StupidScript

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 222 posted 6:15 pm on Nov 13, 2004 (gmt 0)

I feel ya, walkman.

MSNbot sucked up 17.5 gigabytes from our server in May, before I had to ban it. Nothing in the SERPs except our domain name (so the index is aware of us but have no data). We feel brutalized.

I just un-banned it and submitted the URLs, so we'll see what happens.

markus007

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 222 posted 5:18 am on Nov 14, 2004 (gmt 0)

the bot fetches 10's of thousands of pages every week but for some reason i don't have more then 6k in the tech preview.

RobBroekhuis

10+ Year Member



 
Msg#: 222 posted 2:48 pm on Nov 15, 2004 (gmt 0)

Same here - MSNbot has been very good at crawling my site for months now. The pages come up at the beta site on a site:myhobbysite.com search, but not on any keyword search (and trust me, I got some pretty unique keyword combos).
Rob

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved