Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Redirect and robots.txt

google is spidering pages I though I blocked

         

smokeybarnable

4:35 pm on Jun 16, 2006 (gmt 0)

10+ Year Member



If I redirect page A to page B and block spiders from accessing page A in my robots text. Will this stop the spider from indexing page B or do I need to block the redirected page too in my robots text. Or, should I not redirect page A and hope google stops indexing pages I don't want it to index?

tedster

10:45 pm on Jun 17, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you want a page not to be indexed, you should block that page itself - using robots.txt and/or robots meta tags. I know that sometimes (like now) Google appears to ignore these factors, but I can't imagine this is a long term situation.

Google will grab urls anywhere they can find them, including your redirect code code from PageA to PageB. Even though PageA is prohibited, if the PageB url is available and not prohibited, it certainly can end up in the index.