homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

robots and nav bars
excluding links in nav bars ...

10+ Year Member

Msg#: 471 posted 11:06 am on Oct 20, 2004 (gmt 0)

We are using an Oracle portal implementation so things are a little different. We use google as an internal search engine for our site search. It works really well, except for 1 thing. If the key word is on the side nav bar then all pages with the nav bar are returned. For example a search of jobs will return all correct listing PLUS other articles that have the nav bar because there is an entry of "jobs" in the nav bar. I wanted to include the meta tag but the code for the nav bar is a seperate piece of html that bolts onto the template so the </head> has already passed by the time the nav bar code starts. From what I have read the robots.txt file is unreliable. Any advice would be greatfully received.



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 471 posted 11:38 pm on Oct 29, 2004 (gmt 0)

Well, this might be a legitimate reason to use cloaking:

Detect the HTTP_User_Agent, and if Google, do not include the nav bar. You'll also want to use the robots-nocache meta-tag to prevent these modified pages from appearing in Google's cache.

It also might be a good idea to leave an html comment in each page briefly explaining why the page served to Google is different from what a user sees, in case you're in a competitive market where a competitor might report you for cloaking... Just a thought.

Robots.txt won't be of much help here, for the reasons you cite and others. Robots.txt works at the URL-level, and not at the "file" level, so it won't help to disallow the files you include in each page on the server side.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved