homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Stopping robots visiting single dynamic page.

10+ Year Member

Msg#: 6 posted 6:29 pm on Apr 21, 2003 (gmt 0)


Hi, my site works on a MVC design i.e all requests go to /controller?requestType=theRequest

e.g. to get home page href="/controller?requestType=home"
to get links page href="/controller?requestType=links"

I now want to block robots going to some pages (eg a home link), so in my robots.txt:

User-agent: *
Disallow: /controller?requestType=home

Does the robots.txt standard and robots understand the parameter or will all requests to /controller be blocked?

Also, if i have links to external websites do they need to be in the robots.txt, or do the robots not traverse them by default?

Thank you



10+ Year Member

Msg#: 6 posted 2:02 am on Apr 22, 2003 (gmt 0)

Hi donovana,

I don't believe the robots protocol can handle dynamic pages. However, I solved this problem on my website by using the mod_rewrite engine.

RewriteRule /foo/random.html$ /foo/?section=random [L]

In other words, any request for /foo/random.html will get redirected to the dynamic page without the browser (or spider) knowing the difference. Then I changed all my links pointing to the dynamic page to point to the static page and put the static page in my robots.txt.

If you want help on mod_rewrite, do a search here at WW for lots of information.

As for external links, your robots.txt applies only to your own domain.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved