homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Why Google is following my links?

5+ Year Member

Msg#: 3861190 posted 7:09 pm on Mar 2, 2009 (gmt 0)

I've following links blocked in my robots.txt:
User-agent: *
Disallow: /tools/email.php
Disallow: /tools/print.php
Disallow: /tools/save.php
Disallow: /*?
Sitemap: http://www.example.com/sitemap.xml.gz

On each page email link is as follows:
<a href="/tools/email.php?aid=100" rel="nofollow">Email this page to a friend</a>

But log shows that Google and other search engines are still following the links. Am I doing something wrong here?




WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3861190 posted 1:37 pm on Mar 3, 2009 (gmt 0)

Your robots.txt should be blocking these urls:

It will allow Google to access a url like this:

Also please remember that rel="nofollow" was not intended to restrict robot access (even though it is called nofollow). Its purpose is to identify untrusted links to Google.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved