Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Robots.txt Stopping robots visiting single dynamic page. donovana
Hi, my site works on a MVC design i.e all requests go to /controller?requestType=theRequest
e.g. to get home page href="/controller?requestType=home"
to get links page href="/controller?requestType=links"
I now want to block robots going to some pages (eg a home link), so in my robots.txt:
Does the robots.txt standard and robots understand the parameter or will all requests to /controller be blocked?
Also, if i have links to external websites do they need to be in the robots.txt, or do the robots not traverse them by default?
I don't believe the robots protocol can handle dynamic pages. However, I solved this problem on my website by using the mod_rewrite engine.
RewriteRule /foo/random.html$ /foo/?section=random [L]
In other words, any request for /foo/random.html will get redirected to the dynamic page without the browser (or spider) knowing the difference. Then I changed all my links pointing to the dynamic page to point to the static page and put the static page in my robots.txt.
If you want help on mod_rewrite, do a search here at WW for lots of information.
As for external links, your robots.txt applies only to your own domain.