Forum Moderators: goodroi

Message Too Old, No Replies

disallow all : including host

         

moua

2:43 am on Aug 22, 2005 (gmt 0)

10+ Year Member



On a domain name
subdomain.domain.com i have made this :
Robots.txt :
User-Agent: *
Disallow: /
Disallow:

The website is not crawled by google.

However, when i search for "domain.com" in google i found a link to subdomain.domain.com (without any informations like title or content).

How to, with robots.txt, exclude google to make any link to this private subdomain, including "/" itself?

Thanks

encyclo

1:26 pm on Aug 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to the forums, moua. Google is behaving correctly in that it has noted a link to your subdomain, but it has not fetched the page as the robots.txt does not permit it. However, it will continue to show the URL as a page which exists.

Simply excluding robots does not make a page or subdomain private - for that, you should be looking at password-protecting or otherwise controlling access to that page or subdomain.

moua

3:52 pm on Aug 22, 2005 (gmt 0)

10+ Year Member



In fact this subdomain can be accessed from anyone, it's public. But i don't want it to be on google.

Someone pointed a link to this subdomain somewhere (a forum) and now google crawl it.

Hum... i think the only way is to ban google with .htaccess
Do you know wich rules should i use?

encyclo

3:57 pm on Aug 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just noticed - your robots.txt should read:

User-Agent: *
Disallow: /

without the second Disallow line, which may well confuse the issue. :)