---- Pages are indexed even after blocking in robots.txt
shaunm - 7:07 am on Sep 3, 2012 (gmt 0)
@tedster Thank you so much!
Google then constructs a title and snippet for the URL just from references rather than by crawling the page directly.
But it doesn't look like a snippet, it just looks how other pages are displayed in Google's search results. I have have come through this snippet stuff for other websites though.
And remember to change your robots.txt file so you now ALLOW googlebot to crawl the page. Unless they crawl, they won't ever read the robots meta tag.
Oops, I wouldn't have done that if you hadn't informed me. Thanks a ton!
@not2easy Thanks buddy!
If the URL is in your sitemap, the page will be crawled.
Are you sure that even though I may block a web page using noindex meta tags, the page will still be indexed if the URL has been included in the SITEMAP?!? Because I have never heard of this before. Can you give me some references or share your personal experiences? THanks
@aakk9999 Thanks for that mate! I will keep those things in mind as well :)
@tedster Thanks again, ted :)
@atlrus Thank you!
@Robert Thank you for replying on my post and for the provided URL reference :)