Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi


Stopping robots visiting single dynamic page.

6:29 pm on Apr 21, 2003 (gmt 0)

10+ Year Member


Hi, my site works on a MVC design i.e all requests go to /controller?requestType=theRequest

e.g. to get home page href="/controller?requestType=home"
to get links page href="/controller?requestType=links"

I now want to block robots going to some pages (eg a home link), so in my robots.txt:

User-agent: *
Disallow: /controller?requestType=home

Does the robots.txt standard and robots understand the parameter or will all requests to /controller be blocked?

Also, if i have links to external websites do they need to be in the robots.txt, or do the robots not traverse them by default?

Thank you

2:02 am on Apr 22, 2003 (gmt 0)

10+ Year Member

Hi donovana,

I don't believe the robots protocol can handle dynamic pages. However, I solved this problem on my website by using the mod_rewrite engine.

RewriteRule /foo/random.html$ /foo/?section=random [L]

In other words, any request for /foo/random.html will get redirected to the dynamic page without the browser (or spider) knowing the difference. Then I changed all my links pointing to the dynamic page to point to the static page and put the static page in my robots.txt.

If you want help on mod_rewrite, do a search here at WW for lots of information.

As for external links, your robots.txt applies only to your own domain.


Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month