Welcome to WebmasterWorld Guest from 50.16.107.222

Forum Moderators: mack

Message Too Old, No Replies

Error: MSN creating it's own links on my site to crawl?

Home section pages aren't in my mail section!

     
4:45 am on Jul 1, 2006 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 26, 2004
posts:3162
votes: 18


I've seen the search spiders crawling so much that they've gotten bored and decided to create their own addresses to crawl and thus flooding my 404 reports with rubbish I've got to dig through to find actual 404s that visitors are encountering.

65.54.188.99 - - [01/Jul/2006:03:43:34 +0000] "GET /mail/home-music.php HTTP/1.0" 404 12858 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"

Mail and Home are two seperate sections on my site. I'm assuming MSN dude has some connection to MSN? I dno't do anything too dynamic to my links to confuse the crawlers.

- John

8:07 pm on July 2, 2006 (gmt 0)

Full Member

10+ Year Member

joined:May 23, 2004
posts:261
votes: 0


Sorry about that. We'll have a look at it. If you have more specifics you can send me, please do so. (Probably need to use a "sticky" due to rules against URLs in the forums.)

Thanks for letting us know.

9:24 pm on July 2, 2006 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 26, 2004
posts:3162
votes: 18


Not a problem and not as obnoxious as the few dozen or couple hundred 404s generated by Ask's spider looking for urls that have nothing to do with my site to begin with! I'll sticky you the info and thanks for keeping an eye out. :)

- John

8:13 am on July 19, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 2, 2005
posts:505
votes: 0


i just spotted msn looking for mysite.com/?=682 (which has NEVER existed)... anyone ever figure out what's up with this?
12:51 pm on July 19, 2006 (gmt 0)

Full Member

10+ Year Member

joined:May 23, 2004
posts:261
votes: 0


Can you get me the log entry? Which MSN bot was it?