Welcome to WebmasterWorld Guest from 34.237.76.249

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and password-protection

     
10:43 pm on Nov 15, 2006 (gmt 0)

New User

10+ Year Member

joined:June 10, 2002
posts:32
votes: 0


A stupid question really... My client has a testing server (subdomain.example.com) that is password-protected. When I was doing a backlink check for example.com, I've noticed that subdomain.example.com was listed there, hence it can probably be found through other means as well. I do not want the subdomain to appear at all (or at least appear as little as humanly possible). There is a robots.txt in place, but again, since the whole subdomain is password-protected, does that mean that SEs are not able to read that robots.txt file? And if so, what should be done to make it visible to SEs?

Thanks much in advance!

4:45 pm on Nov 16, 2006 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3488
votes: 378


hi hwy2hel,

the search engines are probably seeing your home page which has the username and password input screen becuase that page is not password protected. if your subdomain is properly password protected than the bots will not be able to crawl it.

if you want a more aggressive approach to restricting access you probably will want to make changes to your .htaccess file

10:56 pm on Nov 16, 2006 (gmt 0)

New User

10+ Year Member

joined:June 10, 2002
posts:32
votes: 0


thanks much! this helps :)