Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Is Robots.txt Blocking Bots? Click Tracking


goodroi - 1:36 pm on Jun 10, 2009 (gmt 0)


Robots.txt is a voluntary protocol. The big search engines honor it and will follow it. Some smaller bots (including scraper bots) will ignore it. If you look in your log file you should be able to see what is behind those clicks.


Thread source:: http://www.webmasterworld.com/robots_txt/3929850.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com