Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- files requested by robots.txt


jdMorgan - 9:37 pm on Jun 7, 2003 (gmt 0)


Andrea,

Welcome to WebmasterWorld [webmasterworld.com]!

It has been rumored that visiting a page with the Google Toolbar installed can cause a visit from the Googlebot. Because of this, I'd recommend you add the <meta name="robots" content="noindex"> tag to those pages you absolutely don't want disclosed.

If you'd like to write a more compact robots.txt, I posted some suggestions here [webmasterworld.com].

Jim


Thread source:: http://www.webmasterworld.com/robots_txt/28.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com