homepage Welcome to WebmasterWorld Guest from 54.226.173.169
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt for yahoo and google.com
google and yahoo robots.txt
adisakmai




msg:1526877
 3:27 am on Jun 10, 2004 (gmt 0)

I don't want yahoo.com index whole website, and I don't want every spider index JAVA files.

Is this correct?
----------------------------------------------------------------

User-agent: Slurp
Disallow: /

User-agent: *
Disallow: /*.js$

-----------------------------------------------------------------

Many Thank

 

Abdelrhman Fahmy




msg:1526878
 4:18 am on Jun 10, 2004 (gmt 0)

Correct, and you may look at
[help.yahoo.com...]

finally some one who don't want Yahoo to index his website! :)

jdMorgan




msg:1526879
 4:47 am on Jun 10, 2004 (gmt 0)

Only Google will recognize the "wildcard" *.js -- It is not standard syntax.

The standard robots.txt uses prefix matching, which means that you will have to disallow each .js file individually, or place them all in a subdirectory and disallow that subdirectory.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved