| 12:47 pm on Jul 8, 2014 (gmt 0)|
welcome to WebmasterWorld, sathishvaiha!
if you want to "stop Google from indexing a page" you should use meta robots noindex element.
using robots.txt will exclude a bot from crawling but is not an indexing directive.
| 6:39 pm on Jul 8, 2014 (gmt 0)|
Technical answer: Googlebot does recognize the "Allow" directive, so what you want to do is theoretically possible. You Disallow the directory, and then Allow its individual pages.
But a "noindex" meta is probably a better way to go-- especially when you're talking about rewritten URLs where the directories don't physically exist.
| 4:19 am on Jul 9, 2014 (gmt 0)|
Awesome replies! Thanks @phranque & @lucy24.
| 2:55 am on Sep 12, 2014 (gmt 0)|
Sadly, you will have to let the G see the page to see the directives. the only way to keep G from seeing a page is to password protect it.
Aside: don't rely on noindex to keep G from reading the page. they have to to see noindex. Catch-22
Have to ask: Why do you want to disallow G for that page? (General response, do not be too specific)
| 4:55 am on Oct 8, 2014 (gmt 0)|
You may Stop Google crawler to index your website page by using the Robot.txt File. You may need to place this file in root of your website.
| 7:23 am on Oct 9, 2014 (gmt 0)|
|You may Stop Google crawler to index your website page by using the Robot.txt File |
| 10:01 pm on Oct 9, 2014 (gmt 0)|
Clarification re: robots.txt:
Only good bots will honor, but even then your doc might be visible from other links/locations. If you don't want goggle, or anyone, to see that page, password protect it, ie. don't make it available to the net at large.
ps: if you don't want g to see it, don't put it up in the first place. They will find it via other links/references./email, etc. You can't "hide" anything from the Gorg.