homepage Welcome to WebmasterWorld Guest from 54.166.8.138
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt Disallow query
Disallow specific query in Robots.txt
frogz




msg:3822317
 10:45 pm on Jan 8, 2009 (gmt 0)

Hello!

How would I go about Disallowing a query such as:

[domain.com...]

the numbers of course will change, I would like all "post_id-*.html"
queries Disallowed in robots.txt.

any suggestions?

thnx in advance!

 

frogz




msg:3822500
 4:29 am on Jan 9, 2009 (gmt 0)

bump

g1smd




msg:3822655
 12:35 pm on Jan 9, 2009 (gmt 0)

The rule matches "from the left".

More important is to know of similar URLs that you DO NOT wish to disallow.

In your case:

Disallow: /foru

would work, but would obviously disallow a load of other things that you probably do not want to disallow.

You talk about "post_id-*.html" but is there anything you *do* want to allow in the form "post_id-*.<something else>"?

Perhaps:

Disallow: /forum/post_id

is all you need.

That will match any URL where the URL begins with /forums/post_id.

frogz




msg:3823138
 10:35 pm on Jan 9, 2009 (gmt 0)

Thank you for taking the time to reply g1smd!

Actually the post_id should work fine, there is a topic_id as well that lists the exact same content, only threaded, instead of individual posts.

again thank you again for your time. In due time, I am sure I will subscribe to this site, It's very helpful.

regards, frogz

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved