Forum Moderators: martinibuster
If your server is being crawled by Google and the other search engines on the HTTPS server it slows down your servers and also is duplicate content because the HTTP and HTTPS servers are considered separate domains.
I just had to fight tooth and nail to get Google, Bing and Yahoo to stop crawling my HTTPS server and it was *NOT* fun.
Google responded properly to a robots.txt file just on the HTTPS server telling them not to crawl that server whatsoever.
Both MSNBot (BING) and Yahoo! Slurp didn't follow the HTTPS robots.txt properly and required directly contacting both companies.
Bing quickly fixed the problem but Yahoo is still not responding to my technical requests or the robots.txt file.