Forum Moderators: phranque
I end up with a list of pages like
[somedomain.com...]
[somedomain.com...]
[somedomain.com...]
[somedomain.com...]
etc etc
Is there a way to set a rule in robots.txt to stop it from spidering pages that have PHPSESSID on the query string, while allowing all the other pages to still be spidered?
I'm concerned this could be causing my site to not be spidered as most bots seem to grab a few pages see this, then not return for a month only to have the same thing happen again.
Thanks for the help
ps. I didn't know which forum to put this in, so please move it if need be.