homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robot.txt: Wildcards

5+ Year Member

Msg#: 3960469 posted 6:49 pm on Jul 27, 2009 (gmt 0)

Question for the group:

If I have a url with the following string:

Let's say I want to disallow everything after the query value. Would the following robot.txt disallow be applicable:
User-agent: *
Disallow: /sub1/*/*/*.html?


Any help is much appreciated



WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3960469 posted 12:54 pm on Jul 28, 2009 (gmt 0)

For your robots.txt to block access to all URLs that include a question mark, you can add this entry:

User-agent: *
Disallow: /*?

This is not part of the official robots.txt protocol but it is supported by Google, Yahoo and Bing.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved