Welcome to WebmasterWorld Guest from 54.158.143.40

Forum Moderators: goodroi

Message Too Old, No Replies

External Javascript and bots and robots.txt?

   
4:42 am on Mar 12, 2003 (gmt 0)

10+ Year Member



Maybe Im parinoid but it's just my nature. I want to keep all se spiders from following sorting links on directory pages. It would basically see duplicate content on different urls if it did, just resorted.

So in an external javascript file I have

function locate(page)
{
thePage='http://www.domain.net/' +page
window.location=thePage
}

Then the url's look like this

<a href="#" onclick="locate('page.php'); return false;">Send</a>

Is there any chance that a SE spider would be able to understand that

thePage='http://www.domain.net/' +page
means
[domain.net...]

On a side not, I thought about putting all my css and js files in thier own dir and having a robots.txt file saying not to spider it. What would googlebot and other se spiders think of this? Would they think I was trying to spam with css colored text and such?

6:07 pm on Mar 12, 2003 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Hey Jeremy,

On the first part I don't think spiders will be able to follow the links.

I've recently removed my js and css directorys from my robots.txt dissallow because I found that it disabled cached copies in Alexa and Google. Links couldn't be followed and pages looked like heck. After thinking about it I figured there's no harm in allowing spiders to access them.

Jim