Forum Moderators: Robert Charlton & goodroi
Unindexed result shows like this:
Page Title
Similar page - Note this
Many of them even display '23 hours ago' or whenver the last crawl date was. The thing I can't understand is why Google is crawling these as I have them blocked in my robots.txt.
I have the following setup to block page-1234.html, page-1235.html, etc.
Disallow: /page
Disallow: /page*
Disallow: /page$
According to Google webmaster tools these pages will be blocked. Why are they showing in search results?
Plus you will need to include a noindex tag on each page to make sure that G never indexes it.