Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
My website is example.com, and I have been developing it under beta.example.com
I had hoped search engines wouldn't find that, but they have, so now I need to hide it with robots.txt. But I'm really scared of putting robots.txt in the root of beta.foo.com it will screw up my main site.
So, are the search engines supposed to look at [beta.example.com...] and I put:
Thanks for you help, I gotta get this blocked soon!
[edited by: goodroi at 12:50 pm (utc) on Aug. 28, 2009]
[edit reason] Examplified [/edit]
Search engine robots expect <any-subdomain>.example.com and example.com to be separate Web sites, each with it's own unique robots.txt and unique pages, so your approach is the correct one. It when these requests are *not* handled separately by a server that you can get into trouble -- typically because of duplicate-content issues.