Forum Moderators: goodroi
When I do a search using the code "site:example.com" on my domain, I found several links which seems to be not appropriate for Google to index it, according to the one who reviewed my request of reviewing it.
That particular person has suggested me to block some files' link in my wordpress blog on my domain.
Examples of those links :
1) http://www.example.com/blog/feed
2) http://www.example.comblog/wp-login.php
3) http://www.example.com/wp-login.php?action=lostpassword
4) http://www.example.com/blog/wp-login.php?action=register
How should I block those links from being indexed by Google or any other robots ?
I do know how to block a folder from being index, but I do not know how to block a link from being indexed.
Thanks in advance for your help.
If you only care about blocking Google then use pattern matching (which is not officially part of robots.txt, but is supported by Google).
User-agent: *
Disallow: /*feed
Disallow: /*login
That will block googlebot from all urls that contain "feed" or "login".
Ok , thanks for the information given too. But is there any other way of blocking those file's link other than what you've suggested ?
My concern was this : who knows that in the future, I might be posting something call " the secrets of feed burner " which my wordpress will then generate a link - www.example.com/blog/2008/the-secrets-of-feed-burner.html or something like that. In that case, Google bots or some other bots might be blocked from indexing my particular link too, am I correct ?
That is why I was asking if there's any other way to block those files.
Anyway, thanks again for your reply :)
If you want to get fancy and more security you could use a .htaccess file. But I think that could probably be overkill for your situation.
Oh yes, about the particular problem that I've mentioned up there, I found a solution to solve it.
Someone from the Google Group has suggested me to put this code into my robots.txt
============================================================
User-agent: *
Disallow: /blog/wp-login.php
============================================================
That means if I wanted to block a file under /blog/ directory in my hosting account, and the particular file's name is wp-login.php, the clode above can be applied.
