Forum Moderators: open

Message Too Old, No Replies

Make SE spider a subdomain but not root domain?

Subdomain is a completely different IP, not a subdirectory

         

Winooski

8:57 pm on Feb 23, 2003 (gmt 0)

10+ Year Member



Here's the scenario: I have a client with a dynamically-created site for which we developed an essentially mirror static-page version with a subdomain, subdomain.sitename.com. Now I need to do something to prevent SEs (especially the big bad G) from spidering the dynamic site, instead only spidering the subdomain mirror. The subdomain should be the only URL that SEs know about.

www.sitename.com, bad.
subdomain.sitename.com, good. :)

I've been getting familiar with use of robots.txt, thanks in no small part to this excellent robots.txt post [webmasterworld.com], but I haven't figured out how to get an SE to not spider the root, yet spider the subdomain. I should mention that the subdomain is not the typical subdirectory-turned-into-a-subdomain, but instead is a completely different IP address/host/etc.

marcs

6:21 am on Feb 25, 2003 (gmt 0)

10+ Year Member



I should mention that the subdomain is not the typical subdirectory-turned-into-a-subdomain, but instead is a completely different IP address/host/etc.

As these are two different "sites", simply upload a robots.txt file to the domain site blocking all robots. Leave the subdomain wide open to be spidered/indexed.

Winooski

3:09 pm on Feb 25, 2003 (gmt 0)

10+ Year Member



Thanks marcs! In other words, there's no way that the robots.txt file for www.sitename.com can restrict access to subdomain.sitename.com, so I'm all set.