Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Robots.txt for yahoo and google.com

google and yahoo robots.txt

3:27 am on Jun 10, 2004 (gmt 0)

10+ Year Member

I don't want yahoo.com index whole website, and I don't want every spider index JAVA files.

Is this correct?

User-agent: Slurp
Disallow: /

User-agent: *
Disallow: /*.js$


Many Thank

4:18 am on Jun 10, 2004 (gmt 0)

10+ Year Member

Correct, and you may look at

finally some one who don't want Yahoo to index his website! :)

4:47 am on Jun 10, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Only Google will recognize the "wildcard" *.js -- It is not standard syntax.

The standard robots.txt uses prefix matching, which means that you will have to disallow each .js file individually, or place them all in a subdirectory and disallow that subdirectory.



Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved