Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

External Javascript and bots and robots.txt?



4:42 am on Mar 12, 2003 (gmt 0)

10+ Year Member

Maybe Im parinoid but it's just my nature. I want to keep all se spiders from following sorting links on directory pages. It would basically see duplicate content on different urls if it did, just resorted.

So in an external javascript file I have

function locate(page)
thePage='http://www.domain.net/' +page

Then the url's look like this

<a href="#" onclick="locate('page.php'); return false;">Send</a>

Is there any chance that a SE spider would be able to understand that

thePage='http://www.domain.net/' +page

On a side not, I thought about putting all my css and js files in thier own dir and having a robots.txt file saying not to spider it. What would googlebot and other se spiders think of this? Would they think I was trying to spam with css colored text and such?


6:07 pm on Mar 12, 2003 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Hey Jeremy,

On the first part I don't think spiders will be able to follow the links.

I've recently removed my js and css directorys from my robots.txt dissallow because I found that it disabled cached copies in Alexa and Google. Links couldn't be followed and pages looked like heck. After thinking about it I figured there's no harm in allowing spiders to access them.



Featured Threads

Hot Threads This Week

Hot Threads This Month