homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

robots.txt: Disallow: /links.htm
I want to disallow "links.htm"

Msg#: 4067994 posted 1:06 pm on Jan 26, 2010 (gmt 0)

I want to disallow "links.htm" but allow "links.html".

With "Disallow: /links.htm" in robots.txt both "links.htm" and "links.html" are disallowed. How should my robots.txt look like ?



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 4067994 posted 2:30 pm on Jan 26, 2010 (gmt 0)

You can add an "Allow" for "/links.html" after the "Disallow" for "/links.htm", but this will only work for the major search engines which support the "Allow:" extension to the Standard for Robot Exclusion. Many search engines don't support this extension.

If you want a solution that works for all robots, then change the name of one of these pages so that a prefix-match no longer results in a "collision" between the two names.

The prefix-matching behavior of robots.txt must be taken into account when naming resources and directories -- along with access control, cache-control, HTTP protocol requirements (naming restrictions), maintainer privilege levels (Who in your organization has access to maintain which directories?), server performance, site organization, and SEO considerations. Picking a good "name" (a URL) for a resource is not something that should or can be done instantly -- it requires some careful consideration.



Msg#: 4067994 posted 10:34 am on Feb 2, 2010 (gmt 0)

Alternatively you could put a 'noindex,nofollow' meta tag in the links.htm file.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved