Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- And Now Google's Doing It. JS Stats Show GoogleBot


TheMadScientist - 10:52 pm on May 14, 2011 (gmt 0)


I guess the conclusion I have to draw is:

I still haven't seen GoogleBot disregard robots.txt, but Google does with their other user-agents and if you get the UA from a Reverse Look Up, then you might THINK GoogleBot did it, but it might (likely?) have been Google using a different automated 'non-bot' user-agent, so they can throw protocol to the wind when it suits their purposes.

It's wasn't a bot that accessed my disallowed pages...
It was an automated web page grabber they use to show previews!


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4312058.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com