Forum Moderators: rogerd & travelin cat

Message Too Old, No Replies

Need Robots.txt Help to Disallow Just One Category

         

ubound

7:27 am on Sep 4, 2018 (gmt 0)

10+ Year Member Top Contributors Of The Month



I have a little bit of embarrassing problem, but I can't figure out it on my own. For some reason, I don't see the mistake.

So I need to disallow just one (not all) category of my Wordpress website. Let's say the URL is:
siteurl com/category/featured/

What I did in my robots.txt is:
User-agent: *
Disallow: /category/featured/

But when tested with Google's robots.txt tool or a free third party tool, they both say that crawling this page is allowed?

Can someone please point out my mistake? Thanks!

ubound

7:44 am on Sep 4, 2018 (gmt 0)

10+ Year Member Top Contributors Of The Month



Found another solution. This can be deleted although I still don't know what I did wrong.

keyplyr

8:00 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Looks correct. Did you visually verify that robot.txt was indeed updated at your server?

Also, one common mistake with a robots.txt tester is including a superfluous slash since the tool may already include it:
//category/featured/

ubound

8:58 am on Sep 4, 2018 (gmt 0)

10+ Year Member Top Contributors Of The Month



Looks correct. Did you visually verify that robot.txt was indeed updated at your server?

Also, one common mistake with a robots.txt tester is including a superfluous slash since the tool may already include it:
//category/featured/


Yes, it was updated in Search Console. The third party tool also verified that it used the new version. The slash was OK.

keyplyr

10:38 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Well as I said, that Disallow looks correct for what you're intending.

Is there something above that may be cascading and interferring? Did GSC's tool highlight any warnings?

If not, I'm out of ideas. Sorry.

ubound

11:32 am on Sep 4, 2018 (gmt 0)

10+ Year Member Top Contributors Of The Month



I know right? I can't see anything wrong either, but the good thing I noticed something that was in my WP dashboard the whole time -- and it's the option to noindex one category at a time without noindexing others. So the issue is fixed, the rest is mystery!