homepage Welcome to WebmasterWorld Guest from 54.161.166.171
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
External Javascript and bots and robots.txt?
JeremyL




msg:1527784
 4:42 am on Mar 12, 2003 (gmt 0)

Maybe Im parinoid but it's just my nature. I want to keep all se spiders from following sorting links on directory pages. It would basically see duplicate content on different urls if it did, just resorted.

So in an external javascript file I have
function locate(page)
{
thePage='http://www.domain.net/' +page
window.location=thePage
}

Then the url's look like this
<a href="#" onclick="locate('page.php'); return false;">Send</a>

Is there any chance that a SE spider would be able to understand that

thePage='http://www.domain.net/' +page
means
[domain.net...]

On a side not, I thought about putting all my css and js files in thier own dir and having a robots.txt file saying not to spider it. What would googlebot and other se spiders think of this? Would they think I was trying to spam with css colored text and such?

 

jimbeetle




msg:1527785
 6:07 pm on Mar 12, 2003 (gmt 0)

Hey Jeremy,

On the first part I don't think spiders will be able to follow the links.

I've recently removed my js and css directorys from my robots.txt dissallow because I found that it disabled cached copies in Alexa and Google. Links couldn't be followed and pages looked like heck. After thinking about it I figured there's no harm in allowing spiders to access them.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved