homepage Welcome to WebmasterWorld Guest from 54.166.228.100
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Why Google is following my links?
himalayaswater




msg:3861192
 7:09 pm on Mar 2, 2009 (gmt 0)

I've following links blocked in my robots.txt:
User-agent: *
Disallow: /tools/email.php
Disallow: /tools/print.php
Disallow: /tools/save.php
Disallow: /*?
Sitemap: http://www.example.com/sitemap.xml.gz

On each page email link is as follows:
<a href="/tools/email.php?aid=100" rel="nofollow">Email this page to a friend</a>

But log shows that Google and other search engines are still following the links. Am I doing something wrong here?

TIA

 

goodroi




msg:3861690
 1:37 pm on Mar 3, 2009 (gmt 0)

Your robots.txt should be blocking these urls:
http://www.example.com/tools/email.php
http://www.example.com/tools/email.php?aid=100

It will allow Google to access a url like this:
http://www.example.com/red/tools/email.php

Also please remember that rel="nofollow" was not intended to restrict robot access (even though it is called nofollow). Its purpose is to identify untrusted links to Google.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved