Robots.txt

Stopping robots visiting single dynamic page.

[donovana]

Hi,

Hi, my site works on a MVC design i.e all requests go to /controller?requestType=theRequest

e.g. to get home page href="/controller?requestType=home"
to get links page href="/controller?requestType=links"

I now want to block robots going to some pages (eg a home link), so in my robots.txt:

User-agent: *
Disallow: /controller?requestType=home

Does the robots.txt standard and robots understand the parameter or will all requests to /controller be blocked?

Also, if i have links to external websites do they need to be in the robots.txt, or do the robots not traverse them by default?

Thank you

[Finder]

Hi donovana,

I don't believe the robots protocol can handle dynamic pages. However, I solved this problem on my website by using the mod_rewrite engine.

RewriteRule /foo/random.html$ /foo/?section=random [L]

In other words, any request for /foo/random.html will get redirected to the dynamic page without the browser (or spider) knowing the difference. Then I changed all my links pointing to the dynamic page to point to the static page and put the static page in my robots.txt.

If you want help on mod_rewrite, do a search here at WW for lots of information.

As for external links, your robots.txt applies only to your own domain.