Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- External Javascript and bots and robots.txt?


jimbeetle - 6:07 pm on Mar 12, 2003 (gmt 0)


Hey Jeremy,

On the first part I don't think spiders will be able to follow the links.

I've recently removed my js and css directorys from my robots.txt dissallow because I found that it disabled cached copies in Alexa and Google. Links couldn't be followed and pages looked like heck. After thinking about it I figured there's no harm in allowing spiders to access them.

Jim


Thread source:: http://www.webmasterworld.com/robots_txt/96.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com