Andem - 8:42 pm on May 1, 2012 (gmt 0)
>> you should always implement HTTP Basic Authentication
Thanks for the tip! I've now implemented that. Something to stop Google from crawling everything and sending me traffic.
>> no it not get ignored, its a door for crawlers where only we can restrict the crawlers for some specific contents.
I don't understand.