Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
How to stop Google from indexing a page? I need to block the following page alone from Google Index: sathishvaiha I need to block the following page alone from Google Index through Robots.txt: h t t p://stores.example.com/BlueWidgets I don't want to make Google to stop crawling the following pages. h t t p://stores.example.com/BlueWidgets/Mobile-Phones/Model1 h t t p://stores.example.com/BlueWidgets/Mobile-Phones/Model2 Need your suggestion? Thanks in Advance! [ edited by: engine at 9:48 am (utc) on Jul 8, 2014] [edit reason] obfuscated [/edit]
welcome to WebmasterWorld, sathishvaiha! if you want to "stop Google from indexing a page" you should use meta robots noindex element. using robots.txt will exclude a bot from crawling but is not an indexing directive.
Technical answer: Googlebot does recognize the "Allow" directive, so what you want to do is theoretically possible. You Disallow the directory, and then Allow its individual pages. But a "noindex" meta is probably a better way to go-- especially when you're talking about rewritten URLs where the directories don't physically exist. sathishvaiha
Awesome replies! Thanks @phranque & @lucy24. tangor
Sadly, you will have to let the G see the page to see the directives. the only way to keep G from seeing a page is to password protect it. Aside: don't rely on noindex to keep G from reading the page. they have to to see noindex. Catch-22 Have to ask: Why do you want to disallow G for that page? (General response, do not be too specific)
You may Stop Google crawler to index your website page by using the Robot.txt File. You may need to place this file in root of your website. lucy24
You may Stop Google crawler to index your website page by using the Robot.txt File Nooo .... tangor
Clarification re: robots.txt: Only good bots will honor, but even then your doc might be visible from other links/locations. If you don't want goggle, or anyone, to see that page, password protect it, ie. don't make it available to the net at large. ps: if you don't want g to see it, don't put it up in the first place. They will find it via other links/references./email, etc. You can't "hide" anything from the Gorg.