Forum Moderators: goodroi

Message Too Old, No Replies

I want to disallow extra for Slurp

is this syntax right?

         

baze22

12:37 pm on Apr 19, 2005 (gmt 0)

10+ Year Member



Here are 3 (trimmed down) versions of my robots.txt file:

Initial robots.txt
User-agent: *
Disallow: /forum/member.php
Disallow: /gallery/
====================
modification 1
User-agent: Slurp/2.0
Disallow: /forum/printthread.php
Disallow: /forum/archive/
User-agent: *
Disallow: /forum/member.php
Disallow: /gallery/
====================
modification 2 (current)
User-agent: Slurp/2.0
Disallow: /forum/printthread.php
Disallow: /forum/archive/
Disallow: /forum/member.php
Disallow: /gallery/
User-agent: *
Disallow: /forum/member.php
Disallow: /gallery/

My initial version worked fine. However, earlier this month Yahoo trimmed the number of indexed pages for my site from 12000 to < 200 (yet still spidering 10000+ so far this month). The only thing I can think of might be dup content.

So in an effort to reduce that, I made modification 1 thinking it would disallow the new things for slurp and still disallow the other stuff for slurp and all the others, as well.

This morning, I see slurp pulling member.php and gallery stuff and think to myself "I must have something wrong". So I've made modification 2.

Do I have it right now?

Thanks,

baze

Span

12:55 pm on Apr 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would try User-agent: Slurp without the slash and version number.
[help.yahoo.com ]