Forum Moderators: goodroi
Will it block the search spiders from indexing all the old urls
that currently look like this:
[mydomain....] com/index.php?l=product_list&c=19
Does anyone have experience with this?
Basically I am trying to avoid listing all these old urls in my robots txt file. Make it awfully large with 3000+.
Please remember that it is not officially part of the robots.txt protocol so the smaller bots will probably not follow the wildcard rules.
sidenote - if you are using mod rewrite than you might not even need to use robots.txt wildcards. by 301 redirecting all requests for urls with "?" into static url versions you wouldn't need this rule. having the rule wouldn't hurt either.