Page is a not externally linkable
- WebmasterWorld
-- New To Web Development
---- Spider Link Following


Dreamquick - 2:01 pm on Jan 14, 2003 (gmt 0)


Even if cgi-bin is blocked by robots.txt, Google will read the following link to bar.com

I agree it will read/note the link and potentially include it in the index but are you saying it will access the redirecting script (which it thinks points to "bar.com") violating robots.txt in the process?

- Tony


Thread source:: http://www.webmasterworld.com/new_web_development/253.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com