Welcome to WebmasterWorld Guest from 54.196.175.173

Forum Moderators: mack

Message Too Old, No Replies

Error: MSN creating it's own links on my site to crawl?

Home section pages aren't in my mail section!

     

JAB Creations

4:45 am on Jul 1, 2006 (gmt 0)

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I've seen the search spiders crawling so much that they've gotten bored and decided to create their own addresses to crawl and thus flooding my 404 reports with rubbish I've got to dig through to find actual 404s that visitors are encountering.

65.54.188.99 - - [01/Jul/2006:03:43:34 +0000] "GET /mail/home-music.php HTTP/1.0" 404 12858 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"

Mail and Home are two seperate sections on my site. I'm assuming MSN dude has some connection to MSN? I dno't do anything too dynamic to my links to confuse the crawlers.

- John

msndude

8:07 pm on Jul 2, 2006 (gmt 0)

10+ Year Member



Sorry about that. We'll have a look at it. If you have more specifics you can send me, please do so. (Probably need to use a "sticky" due to rules against URLs in the forums.)

Thanks for letting us know.

JAB Creations

9:24 pm on Jul 2, 2006 (gmt 0)

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Not a problem and not as obnoxious as the few dozen or couple hundred 404s generated by Ask's spider looking for urls that have nothing to do with my site to begin with! I'll sticky you the info and thanks for keeping an eye out. :)

- John

jake66

8:13 am on Jul 19, 2006 (gmt 0)

5+ Year Member



i just spotted msn looking for mysite.com/?=682 (which has NEVER existed)... anyone ever figure out what's up with this?

msndude

12:51 pm on Jul 19, 2006 (gmt 0)

10+ Year Member



Can you get me the log entry? Which MSN bot was it?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month