Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google and robots.txt

Google does not care for our robots.txt

         

atlrus

8:05 pm on Dec 21, 2006 (gmt 0)

10+ Year Member



Our situation:

We redirect to our affiliate partners through pages with META redirect, and we keep all the redirect pages in a folder which we have disallowed in the robots.

Now, all of those pages show when I do s "site:" command - this is understandable, as G have always said that they would include pages like that (from the robots.txt) if you have enough links to them.

But - it seems that Google actually visits those pages, as we have a few that redirect to a french partner - and on the "site:" search, there is a [Translate] option to the french redirects - and there is no way G would know this page goes to a french partner - unless it visits the page and follows it.

I am wrong to think that? Is there any other way Google would know that this page goes to a french one (the partner site is a dot com domain)?