help.robots.txt/external link

disallow external link

[dennis oppelt]

HI
Can I disallow an external link in a robots.txt file?

like:

disallow: http://www.example.com

?

this is because google told me that I could prevent being penalized if I used a robots.txt file to let google think I do not
have a mirror site, which is a duplicate site on a different domain and server and I have a hot link to it on my main site

thanks
Dennis

[Dijkgraaf]

You can't disallow external links in robots.txt.
What you can do have is the following
<a href="http://www.example.com" rel="nofollow">something</a>

But you don't need to do that, all you have to do is on your mirror site have a robots.txt that has

disallow: /
which says, you aren't allowed to request anything from that domain.

[dennis oppelt]

thanks

but I cannot disallow the mirror site to be indexed because we want to submit the mirror site to search engines also, as a seperate site?

so maybe that special html code you mention will work to prevent me from being penalized?

PS: If I use a robots.txt file

does it override any meta tags I may have such as : 'index follow'

or should I delete those and stick to just one?

thanks alot

Dennis

[Dijkgraaf]

Well if you want both sites to be indexed, then you need to make sure their contents are unique.

robots.txt stops them requesting a resource altogether, if it is disallowed there it doesn't even get to read the meta tags in the page, as it shouldn't even request it.

The only downside with robots.txt is that sometimes they will have a URL only listing for a disallowed page, but as this only comes up when you do a search for all pages in a site, not a biggey in my opionion.

If you have both robots.txt and meta tags, you are doubly coverered.

[dennis oppelt]

thanks

one last thing.

Google told me (implied?) that if I do not have any link (or block it from indexing) on my main site to my other duplicate site that is on a different server and domain, that google would not penalize me cause they would not see it as a mirror site.
I cannot afford to be blacklisted

is having a mirror site safe or not?

thanks alot
Dennis

[Dijkgraaf]

If you were to stop them indexing the mirror site, there shouldn't be any issue.

I don't think the link matters as long you have the mirror site blocked from being indexed.

The only penalty that would happen if you didn't block one of them from being indexed is that one of them wouldn't show up in the results, which one it would show would depend on what Google decided is the original.

Why do you need a mirror site anyway?

[dennis oppelt]

we don't turns out that my boss only wanted the new domain to point to the old site

so I am deleting the second site.

funny thing, maybe I am crazy, but I tried one of those 'robots.txt' validators?

(seems everyone has the same script on their site for that) and I typed in a disallow code 2 different ways:

disallow: /styles
and
disallow: \styles

I did them seperately
both times it told me that the code had no erros?

should not the backslash be wrong?

thxs

dennis