Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- msnbot/2 snapshots?

caribguy - 7:34 pm on Sep 19, 2009 (gmt 0)


Spot the difference :)

www.example.com 65.55.106.nnn - - [19/Sep/2009:14:14:37 -0500] "GET /widget HTTP/1.1" 200 16511 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
www.example.com 65.55.106.nnn - - [19/Sep/2009:14:14:46 -0500] "GET /widget HTTP/1.0" 200 85960 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

HTTP/1.1 vs HTTP/1.0 the former uses .gz

Thread source:: http://www.webmasterworld.com/search_engine_spiders/3992111.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com