homepage Welcome to WebmasterWorld Guest from 54.198.42.105
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Robots.txt it odd
different robots txt shown to browser
nippi




msg:710275
 6:55 am on Apr 11, 2005 (gmt 0)

Has anyone ever seen a site where the pages presented to google are index no follow for the robots meta tag, but index follow when presented to a browser?

Or, and instance where a different robots.txt is presented to google, than to a browser?

 

Wizard




msg:710276
 6:20 pm on Apr 11, 2005 (gmt 0)

Of course it's possible, and in most cases it would be a form of cloaking.

jomaxx




msg:710277
 6:33 pm on Apr 11, 2005 (gmt 0)

I don't see how you could ever know if the robots.txt was cloaked.

g1smd




msg:710278
 6:55 pm on Apr 11, 2005 (gmt 0)

Why would you even bother to show a robots.txt file to a browser?

They make no use of it at all.

Wizard




msg:710279
 7:43 pm on Apr 11, 2005 (gmt 0)

I don't see how you could ever know if the robots.txt was cloaked.

If I can see different robots.txt using normal user-agent header and different if using "googlebot" user agent, that's a prove of cloaking, and I understand that's what nippi's talking about.

Why would you even bother to show a robots.txt file to a browser?

To confuse your competition, obviously.

g1smd




msg:710280
 7:50 pm on Apr 11, 2005 (gmt 0)

Why not show nothing to the browser?

Even less work. Looks like there is no robots file at all.

encyclo




msg:710281
 8:04 pm on Apr 11, 2005 (gmt 0)

I think the original paster was referring to robots meta tags rather than a robots.txt file: in particular, cloaking to add a
<meta name="robots" content="index,nofollow"> only for Googlebot et al. All this probably for a links page to "convert" link exchanges into one-way links.

If the cloaking is done properly, it is difficult to detect - assuming that there is a noarchive tag as well, otherwise you can see it in the Cache link in the serps.

Wizard




msg:710282
 9:10 pm on Apr 11, 2005 (gmt 0)

I think the original paster was referring to robots meta tags rather than a robots.txt

I have seen both, and both may be used in similiar purposes.

All this probably for a links page to "convert" link exchanges into one-way links.

You're right.

If the cloaking is done properly, it is difficult to detect - assuming that there is a noarchive tag as well, otherwise you can see it in the Cache link in the serps.

But seeing no cache for links page of you linking partner may be red flag itself - he's hiding something! When I was using cloaking to stuff keywords in the past, in times when it used to work, I always used 'noarchive' tag in order to prevent people from seeing how cloaked page appeared.

jomaxx




msg:710283
 9:49 pm on Apr 11, 2005 (gmt 0)

Wizard, showing a different robots.txt based on the user agent alone would indeed be proof of cloaking, but that won't happen if the cloaking is competently done. In that case, as far as I can see the cloaking would be undetectable to everyone except Google.

[Added: ...unless the robots.txt itself gets indexed, which from a quick search in Google can sometimes happen.]

larryhatch




msg:710284
 5:58 am on Apr 12, 2005 (gmt 0)

An important point you all made!

Lets say some site hogs PR by NOT showing outbound links to G and Y,
but DOES show them to anybody browing. A cloaked links page say.

Is there some way I can detect this? - Larry

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved