If Robots.txt and noindex, nofollow aren't enoough. . .
9:54 am on Oct 8, 2010 (gmt 0)
. . . What are people's thoughts on redirecting users where a cookie cannot bet set, thus redirecting even the most aggressive crawlers?
Mildly grey hat?
12:34 pm on Oct 11, 2010 (gmt 0)
I know many people that redirect users when cookie can not be set. You can send them to a screen asking them to enable cookies or give them a captcha to prove they are not a bot. The main problem with this strategy is that you can end up blocking search engines from crawling your website if you are not careful.