Forum Moderators: goodroi
Initial robots.txt
User-agent: *
Disallow: /forum/member.php
Disallow: /gallery/
====================
modification 1
User-agent: Slurp/2.0
Disallow: /forum/printthread.php
Disallow: /forum/archive/
User-agent: *
Disallow: /forum/member.php
Disallow: /gallery/
====================
modification 2 (current)
User-agent: Slurp/2.0
Disallow: /forum/printthread.php
Disallow: /forum/archive/
Disallow: /forum/member.php
Disallow: /gallery/
User-agent: *
Disallow: /forum/member.php
Disallow: /gallery/
My initial version worked fine. However, earlier this month Yahoo trimmed the number of indexed pages for my site from 12000 to < 200 (yet still spidering 10000+ so far this month). The only thing I can think of might be dup content.
So in an effort to reduce that, I made modification 1 thinking it would disallow the new things for slurp and still disallow the other stuff for slurp and all the others, as well.
This morning, I see slurp pulling member.php and gallery stuff and think to myself "I must have something wrong". So I've made modification 2.
Do I have it right now?
Thanks,
baze