Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Pages are indexed even after blocking in robots.txt


shaunm - 7:07 am on Sep 3, 2012 (gmt 0)


@tedster
Thank you so much!
Google then constructs a title and snippet for the URL just from references rather than by crawling the page directly.
But it doesn't look like a snippet, it just looks how other pages are displayed in Google's search results. I have have come through this snippet stuff for other websites though.
And remember to change your robots.txt file so you now ALLOW googlebot to crawl the page. Unless they crawl, they won't ever read the robots meta tag.
Oops, I wouldn't have done that if you hadn't informed me. Thanks a ton!

@not2easy
Thanks buddy!
If the URL is in your sitemap, the page will be crawled.
Are you sure that even though I may block a web page using noindex meta tags, the page will still be indexed if the URL has been included in the SITEMAP?!? Because I have never heard of this before. Can you give me some references or share your personal experiences? THanks

@aakk9999
Thanks for that mate! I will keep those things in mind as well :)

@tedster
Thanks again, ted :)

@atlrus
Thank you!

@Robert
Thank you for replying on my post and for the provided URL reference :)


Thread source:: http://www.webmasterworld.com/google/4490125.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com