I'm working on a robots.txt for a new site and have a few questions. I use mod_rewrite to make the URL's look nicer, how, if at all, will that effect the robots.txt? Here's an example:
I have urls of the type:
[to being text only and lg low graphics]
that I would like to keep robots out of because they are printer friendly versions. section1/2, to, lg, article1/2 are not real directories, they are the search engine friendly ones made from the mod_rewrite.
How would most bots interpret something like:
Would they also not spider anything with an URL containing "section1"? The real articles, that I want spidered, are named like:
Maybe it would be best to just specify the /to/ and /lg/ fake directories to be disallowed, but how do I write that, will
On the actual files I also have a :
<meta name="robots" content="noindex, nofollow">
is this sufficient or should I also try to exclude them with the robots.txt and keep the meta tag for those that might ignore the robots.txt?