Welcome to WebmasterWorld Guest from 54.162.157.249

Message Too Old, No Replies

Robots.txt it odd

different robots txt shown to browser

     
6:55 am on Apr 11, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Has anyone ever seen a site where the pages presented to google are index no follow for the robots meta tag, but index follow when presented to a browser?

Or, and instance where a different robots.txt is presented to google, than to a browser?

6:20 pm on Apr 11, 2005 (gmt 0)

10+ Year Member



Of course it's possible, and in most cases it would be a form of cloaking.
6:33 pm on Apr 11, 2005 (gmt 0)

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I don't see how you could ever know if the robots.txt was cloaked.
6:55 pm on Apr 11, 2005 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Why would you even bother to show a robots.txt file to a browser?

They make no use of it at all.

7:43 pm on Apr 11, 2005 (gmt 0)

10+ Year Member



I don't see how you could ever know if the robots.txt was cloaked.

If I can see different robots.txt using normal user-agent header and different if using "googlebot" user agent, that's a prove of cloaking, and I understand that's what nippi's talking about.

Why would you even bother to show a robots.txt file to a browser?

To confuse your competition, obviously.

7:50 pm on Apr 11, 2005 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Why not show nothing to the browser?

Even less work. Looks like there is no robots file at all.

8:04 pm on Apr 11, 2005 (gmt 0)

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I think the original paster was referring to robots meta tags rather than a robots.txt file: in particular, cloaking to add a
<meta name="robots" content="index,nofollow">
only for Googlebot et al. All this probably for a links page to "convert" link exchanges into one-way links.

If the cloaking is done properly, it is difficult to detect - assuming that there is a noarchive tag as well, otherwise you can see it in the Cache link in the serps.

9:10 pm on Apr 11, 2005 (gmt 0)

10+ Year Member



I think the original paster was referring to robots meta tags rather than a robots.txt

I have seen both, and both may be used in similiar purposes.

All this probably for a links page to "convert" link exchanges into one-way links.

You're right.

If the cloaking is done properly, it is difficult to detect - assuming that there is a noarchive tag as well, otherwise you can see it in the Cache link in the serps.

But seeing no cache for links page of you linking partner may be red flag itself - he's hiding something! When I was using cloaking to stuff keywords in the past, in times when it used to work, I always used 'noarchive' tag in order to prevent people from seeing how cloaked page appeared.

9:49 pm on Apr 11, 2005 (gmt 0)

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Wizard, showing a different robots.txt based on the user agent alone would indeed be proof of cloaking, but that won't happen if the cloaking is competently done. In that case, as far as I can see the cloaking would be undetectable to everyone except Google.

[Added: ...unless the robots.txt itself gets indexed, which from a quick search in Google can sometimes happen.]

5:58 am on Apr 12, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



An important point you all made!

Lets say some site hogs PR by NOT showing outbound links to G and Y,
but DOES show them to anybody browing. A cloaked links page say.

Is there some way I can detect this? - Larry

 

Featured Threads

Hot Threads This Week

Hot Threads This Month