homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

deny a spider from visiting a single page
Michael Weir

10+ Year Member

Msg#: 221 posted 9:34 pm on Dec 24, 2003 (gmt 0)


I'm new to experimenting robots.txt files and I want to try something.

I've found that optimizing the content on a site for the most part works effectively across the search engines, except obviously not google - not anymore. :(

I've recently de-optimized a client's index page after it took a hit during the florida update in an attempt to make the content seem more organic and not optimized.

Unfortunately the rankings for that particular page slipped on the other engines. So what I did was make a clone of the index page and named it index2. I plan to reoptimize the index2 page back to the way it was and disallow googlebot visitation rights. Obviously, I don't want to deny googlebot from visiting the rest of the site though.

Is it possible to write a robots.txt file to keep googlebot from visiting just the index2 page?

Or is this just a stupid idea to begin with? Like I said, I have some room for experimenting here so I'm curious. Thanks in advance!




WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 221 posted 10:15 pm on Dec 24, 2003 (gmt 0)


Well, that would be:

User-agent: Googlebot
Disallow: /index2.html

The other alternative is cloaking. Personally, I wouldn't recommend either approach, unless it's a throw-away domain. I'd simply find a way to integrate a second, different, and useful page that would rank well where the other one does not. Others may disagree.

The main problem I see with this two-page robots approach is that you will be splitting your incoming links across two pages. Although the engines other than Google don't use the PageRank concept per-se, some of them do take into account "link popularity" and clustering. So, you'll take a hit from that angle as well. :(


Michael Weir

10+ Year Member

Msg#: 221 posted 3:08 pm on Dec 29, 2003 (gmt 0)

Thanks for your insight Jim! I'll try the robots.txt approach and see what happens.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved