- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- htaccess and banning bots
Key_Master - 11:57 am on May 31, 2013 (gmt 0)
I doubt some of those you have marked as working actually were working. Yandex, Baidu, and Bingbot for example, all use a mozilla user-agent.
|I was using Deny from 168. xxx etc but I was told by the server that it took uo too much processor time. |
That doesn't sound right. I think you should figure out why your previous disallows were crashing the server in the first place.
Thread source: http://www.webmasterworld.com/search_engine_spiders/4579553.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com