- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- MJ12bot
Lord_Majestic - 10:52 am on Jun 24, 2009 (gmt 0)
|In practice it's not doable. |
It's doable if user-agent is set to one of the known bots (like ours) but does not actually obey robots.txt - it would be much harder to apply same logic to faked browser user-agents though.
Thread source: http://www.webmasterworld.com/search_engine_spiders/3937580.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com