Forum Moderators: goodroi
http://www.example.com/?month=-7 and that goes on to /?month=nnnn
Thats when I stopped it. I asume ( sorry, newbie) this is a calendar in wordpress. I have WP in root and a punch of static pages in folders also. I also asume if this crawler does that the Google spider attempts the same and this would be negative (?) for my site. I add in my robots.txt this line:
" Disallow: /?month* " while I understand * is a wildcard and that would stop it but doesn't.
Below is my robots.txt - I got it from a wordpress website saying that would be the best. Any advise for my "problem"?
Thanks for any advise.
User-agent: *
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: */feed
Disallow: /category/*/*
Disallow: */trackback
Disallow: */*/trackback
Disallow: /*?*
Disallow: /*?
Disallow: /?month*
Allow: /wp-content/uploads
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /
# digg mirror
User-agent: duggmirror
Disallow: /
[edited by: encyclo at 12:53 am (utc) on Jan. 13, 2008]
[edit reason] switched to example.com [/edit]