-- Google SEO News and Discussion
---- Downside to noindex?
- 8:10 pm on Mar 17, 2011
If robots.txt stopped Google crawling the page how come it is in Google's index with a proper page title? Thats the bit I don't follow.
I have also seen this and do suspect that on occassion pages are fetched.
At one point last year my WMT data showed so many "Duplicate titles" for the pages stopped by robots.txt that it would be impossible someone linked to each of them with this title.
Anyway, there was a discussion about this in the past [
I also saw this question being asked on Google Webmaster's Forum, but cannot find a link to this right now.
Would you expect that URI stopped via robots.txt is not even requested by the bot? Eg, why should it be requested if the access to it by bots is stopped via robots.txt?
Brought to you by