| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Hit by the Msnbot|
Msnbot? Hmm, I thought it was a myth
My site has finally been crawled by the Msnbot. It gave [search.msn.com...] as the information URL.
It seems to be going quite slow, has anyone had their sites crawled by it yet?
i saw it show up for the first time around the middle of october. it didn't stick around long, i'm wondering what it was looking for :)
Finally saw it run through 1/3 of a site, seems that they are at least getting better at crawling the web...
"better" is much like 'beauty' these days '...in the eye of the beholder'.
You may recall my domain was one of the first ones crawled w-a-a-a-y back when MSN's bots were but fledglings.
However, msnbot/0.11 makes only two requests. One for robots and one for the most obsure page of my entire domain.
It's been doing this every day since it started a week ago.
22.214.171.124 - - [26/Nov/2003:13:38:08 -0800] "GET /robots.txt HTTP/1.0" 200 1524 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
126.96.36.199 - - [26/Nov/2003:13:38:09 -0800] "GET /FileWhereI"Cite"Something.html HTTP/1.0" 200 1432 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
In my readings thru WebmasterWorld, I don't recall seeing anything specifically related to msnbot/0.11 and I'd like to get back to being crawled....regularly.
Anyone seeing similar?
I guess there is the off-chance that was all the bot was/is looking for.....
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved