Forum Moderators: goodroi
I’m on IIS 6 and I need to stop my secure site being crawled with a Robots.txt.
Does anyone know a way of having a separate robots.txt file for my https site.
At the moment I’ve got a duplicate problem as Gbot is indexing my secure site, but the robots.txt file is used by the h*tp:// and https:// so I can’t use
user-agent: *
disallow: /
As it will destroy my non-secure page SERP’s
Anyone have any idea on how I can do this.
Vimes.