---- In the era of increasing numbers of bad bots is robots.txt irrelevant?
Jordo_needs_a_drink - 8:01 pm on Feb 20, 2006 (gmt 0)
What does a typical .htaccess whitelist file look like? Are you allowing access by IP or agent or both? If by agent, is it a correct assumption that you are allowing browsers and certain bots in, and if so, can't bad bots spoof those? If by IP, then how do you keep from blocking normal traffic?