Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

robots. txt file on virtual subdomain

         

epmaniac

11:57 am on Jan 21, 2011 (gmt 0)

10+ Year Member



hi,

i can seem to setup robots.txt for my virtual subdomain...

i want to disallow all bots from it, can anybody help?

furthermore, i think WMT doesnt actually differentiates crawl stats of main domain and virtual subdomains (if set up) because ...if i open my subdomains (which is really slow) the time to download a page of my main domain increases drastically, accompanied by reduced crawl rate.

can someone throw some light on it?

tedster

5:28 pm on Jan 21, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The file should be served whenever subdomain.example.com/robots.txt is requested. To deny all well-behaved bots it should read:

User-agent: *
Disallow: /

How to work that out technically will depend on your server - usually Apache [webmasterworld.com] or Windows IIS [webmasterworld.com]

<added>
The Webmaster Tools issue is a separate issue. Normally Google requests that you verify a subdomain as aseparate "website". If you have registered your website as only "example.com" rather than "www.example.com" I can imagine that there might be a tangle - but the robots.txt Disallow rule should fix that.