homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

External Javascript and bots and robots.txt?

10+ Year Member

Msg#: 96 posted 4:42 am on Mar 12, 2003 (gmt 0)

Maybe Im parinoid but it's just my nature. I want to keep all se spiders from following sorting links on directory pages. It would basically see duplicate content on different urls if it did, just resorted.

So in an external javascript file I have
function locate(page)
thePage='http://www.domain.net/' +page

Then the url's look like this
<a href="#" onclick="locate('page.php'); return false;">Send</a>

Is there any chance that a SE spider would be able to understand that

thePage='http://www.domain.net/' +page

On a side not, I thought about putting all my css and js files in thier own dir and having a robots.txt file saying not to spider it. What would googlebot and other se spiders think of this? Would they think I was trying to spam with css colored text and such?



WebmasterWorld Senior Member jimbeetle us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 96 posted 6:07 pm on Mar 12, 2003 (gmt 0)

Hey Jeremy,

On the first part I don't think spiders will be able to follow the links.

I've recently removed my js and css directorys from my robots.txt dissallow because I found that it disabled cached copies in Alexa and Google. Links couldn't be followed and pages looked like heck. After thinking about it I figured there's no harm in allowing spiders to access them.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved