homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

Error: MSN creating it's own links on my site to crawl?
Home section pages aren't in my mail section!
JAB Creations

 4:45 am on Jul 1, 2006 (gmt 0)

I've seen the search spiders crawling so much that they've gotten bored and decided to create their own addresses to crawl and thus flooding my 404 reports with rubbish I've got to dig through to find actual 404s that visitors are encountering. - - [01/Jul/2006:03:43:34 +0000] "GET /mail/home-music.php HTTP/1.0" 404 12858 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"

Mail and Home are two seperate sections on my site. I'm assuming MSN dude has some connection to MSN? I dno't do anything too dynamic to my links to confuse the crawlers.

- John



 8:07 pm on Jul 2, 2006 (gmt 0)

Sorry about that. We'll have a look at it. If you have more specifics you can send me, please do so. (Probably need to use a "sticky" due to rules against URLs in the forums.)

Thanks for letting us know.

JAB Creations

 9:24 pm on Jul 2, 2006 (gmt 0)

Not a problem and not as obnoxious as the few dozen or couple hundred 404s generated by Ask's spider looking for urls that have nothing to do with my site to begin with! I'll sticky you the info and thanks for keeping an eye out. :)

- John


 8:13 am on Jul 19, 2006 (gmt 0)

i just spotted msn looking for mysite.com/?=682 (which has NEVER existed)... anyone ever figure out what's up with this?


 12:51 pm on Jul 19, 2006 (gmt 0)

Can you get me the log entry? Which MSN bot was it?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved