- Search Engines
-- Search Engine Spider and User Agent Identification
---- Baidu's IPs - Which Are Legit?
- 2:21 pm on Aug 15, 2012
rather than just blocking it with robots.txt (which doesn't seem to work.)
FWIW, robots.txt doesn't block or deny anything, rather, robots.txt is a
to compliant bots to honor your wishes.
htaccess on the other hand, is fully capable of denying access to a variety of visitors, and utilizing a variety of methods and/or criteria.
Brought to you by