homepage Welcome to WebmasterWorld Guest from 54.198.130.203
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to stop Google from indexing a page?
I need to block the following page alone from Google Index:
sathishvaiha




msg:4685936
 9:33 am on Jul 8, 2014 (gmt 0)

I need to block the following page alone from Google Index through Robots.txt:
h t t p://stores.example.com/BlueWidgets

I don't want to make Google to stop crawling the following pages.

h t t p://stores.example.com/BlueWidgets/Mobile-Phones/Model1
h t t p://stores.example.com/BlueWidgets/Mobile-Phones/Model2


Need your suggestion? Thanks in Advance!

[edited by: engine at 9:48 am (utc) on Jul 8, 2014]
[edit reason] obfuscated [/edit]

 

phranque




msg:4685988
 12:47 pm on Jul 8, 2014 (gmt 0)

welcome to WebmasterWorld, sathishvaiha!


if you want to "stop Google from indexing a page" you should use meta robots noindex element.
using robots.txt will exclude a bot from crawling but is not an indexing directive.

lucy24




msg:4686121
 6:39 pm on Jul 8, 2014 (gmt 0)

Technical answer: Googlebot does recognize the "Allow" directive, so what you want to do is theoretically possible. You Disallow the directory, and then Allow its individual pages.

But a "noindex" meta is probably a better way to go-- especially when you're talking about rewritten URLs where the directories don't physically exist.

sathishvaiha




msg:4686266
 4:19 am on Jul 9, 2014 (gmt 0)

Awesome replies! Thanks @phranque & @lucy24.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved