Hi all, I paste my robots.txt below. I think I have a problem I need to take care. When I use my sitemap generator - I use GSite Crawler it crawls for ever this kinf of URL on my site:
http://www.example.com/?month=-7 and that goes on to /?month=nnnn
Thats when I stopped it. I asume ( sorry, newbie) this is a calendar in wordpress. I have WP in root and a punch of static pages in folders also. I also asume if this crawler does that the Google spider attempts the same and this would be negative (?) for my site. I add in my robots.txt this line:
" Disallow: /?month* " while I understand * is a wildcard and that would stop it but doesn't.
Below is my robots.txt - I got it from a wordpress website saying that would be the best. Any advise for my "problem"?
Thanks for any advise.
# Google Image
# Google AdSense
# Internet Archiver Wayback Machine
# digg mirror
[edited by: encyclo at 12:53 am (utc) on Jan. 13, 2008]
[edit reason] switched to example.com [/edit]