homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

4 Wordpress File's Link To Be Blocked
How should I block them?

 5:11 am on May 17, 2008 (gmt 0)

Hi there,

When I do a search using the code "site:example.com" on my domain, I found several links which seems to be not appropriate for Google to index it, according to the one who reviewed my request of reviewing it.

That particular person has suggested me to block some files' link in my wordpress blog on my domain.

Examples of those links :
1) http://www.example.com/blog/feed
2) http://www.example.comblog/wp-login.php
3) http://www.example.com/wp-login.php?action=lostpassword
4) http://www.example.com/blog/wp-login.php?action=register

How should I block those links from being indexed by Google or any other robots ?

I do know how to block a folder from being index, but I do not know how to block a link from being indexed.

Thanks in advance for your help.



 10:48 am on May 17, 2008 (gmt 0)

Welcome to posting on WebmasterWorld!

If you only care about blocking Google then use pattern matching (which is not officially part of robots.txt, but is supported by Google).

User-agent: *
Disallow: /*feed
Disallow: /*login

That will block googlebot from all urls that contain "feed" or "login".


 11:13 am on May 17, 2008 (gmt 0)

Thanks for your welcoming word , goodroi :)

Ok , thanks for the information given too. But is there any other way of blocking those file's link other than what you've suggested ?

My concern was this : who knows that in the future, I might be posting something call " the secrets of feed burner " which my wordpress will then generate a link - www.example.com/blog/2008/the-secrets-of-feed-burner.html or something like that. In that case, Google bots or some other bots might be blocked from indexing my particular link too, am I correct ?

That is why I was asking if there's any other way to block those files.

Anyway, thanks again for your reply :)


 2:33 pm on May 19, 2008 (gmt 0)

Feedburner is one word not two :) but you raise a good point that many people overlook. Another option is to use robots metatag.

If you want to get fancy and more security you could use a .htaccess file. But I think that could probably be overkill for your situation.


 3:54 pm on May 19, 2008 (gmt 0)

Thanks for reminding me about the FEEDBURNER thing ;)

Oh yes, about the particular problem that I've mentioned up there, I found a solution to solve it.

Someone from the Google Group has suggested me to put this code into my robots.txt

User-agent: *
Disallow: /blog/wp-login.php

That means if I wanted to block a file under /blog/ directory in my hosting account, and the particular file's name is wp-login.php, the clode above can be applied.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved