Welcome to WebmasterWorld Guest from 107.22.56.104

Forum Moderators: Robert Charlton & andy langton & goodroi

Message Too Old, No Replies

Robots.txt it odd

different robots txt shown to browser

     
6:55 am on Apr 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 3, 2003
posts:809
votes: 0


Has anyone ever seen a site where the pages presented to google are index no follow for the robots meta tag, but index follow when presented to a browser?

Or, and instance where a different robots.txt is presented to google, than to a browser?

6:20 pm on Apr 11, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


Of course it's possible, and in most cases it would be a form of cloaking.
6:33 pm on Apr 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 6, 2002
posts:4768
votes: 0


I don't see how you could ever know if the robots.txt was cloaked.
6:55 pm on Apr 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Why would you even bother to show a robots.txt file to a browser?

They make no use of it at all.

7:43 pm on Apr 11, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


I don't see how you could ever know if the robots.txt was cloaked.

If I can see different robots.txt using normal user-agent header and different if using "googlebot" user agent, that's a prove of cloaking, and I understand that's what nippi's talking about.

Why would you even bother to show a robots.txt file to a browser?

To confuse your competition, obviously.

7:50 pm on Apr 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Why not show nothing to the browser?

Even less work. Looks like there is no robots file at all.

8:04 pm on Apr 11, 2005 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9074
votes: 6


I think the original paster was referring to robots meta tags rather than a robots.txt file: in particular, cloaking to add a
<meta name="robots" content="index,nofollow">
only for Googlebot et al. All this probably for a links page to "convert" link exchanges into one-way links.

If the cloaking is done properly, it is difficult to detect - assuming that there is a noarchive tag as well, otherwise you can see it in the Cache link in the serps.

9:10 pm on Apr 11, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


I think the original paster was referring to robots meta tags rather than a robots.txt

I have seen both, and both may be used in similiar purposes.

All this probably for a links page to "convert" link exchanges into one-way links.

You're right.

If the cloaking is done properly, it is difficult to detect - assuming that there is a noarchive tag as well, otherwise you can see it in the Cache link in the serps.

But seeing no cache for links page of you linking partner may be red flag itself - he's hiding something! When I was using cloaking to stuff keywords in the past, in times when it used to work, I always used 'noarchive' tag in order to prevent people from seeing how cloaked page appeared.

9:49 pm on Apr 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 6, 2002
posts:4768
votes: 0


Wizard, showing a different robots.txt based on the user agent alone would indeed be proof of cloaking, but that won't happen if the cloaking is competently done. In that case, as far as I can see the cloaking would be undetectable to everyone except Google.

[Added: ...unless the robots.txt itself gets indexed, which from a quick search in Google can sometimes happen.]

5:58 am on Apr 12, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


An important point you all made!

Lets say some site hogs PR by NOT showing outbound links to G and Y,
but DOES show them to anybody browing. A cloaked links page say.

Is there some way I can detect this? - Larry

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members