Welcome to WebmasterWorld Guest from 54.167.157.247

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt for yahoo and google.com

google and yahoo robots.txt

   
3:27 am on Jun 10, 2004 (gmt 0)

10+ Year Member



I don't want yahoo.com index whole website, and I don't want every spider index JAVA files.

Is this correct?
----------------------------------------------------------------

User-agent: Slurp
Disallow: /

User-agent: *
Disallow: /*.js$

-----------------------------------------------------------------

Many Thank

4:18 am on Jun 10, 2004 (gmt 0)

10+ Year Member



Correct, and you may look at
[help.yahoo.com...]

finally some one who don't want Yahoo to index his website! :)

4:47 am on Jun 10, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Only Google will recognize the "wildcard" *.js -- It is not standard syntax.

The standard robots.txt uses prefix matching, which means that you will have to disallow each .js file individually, or place them all in a subdirectory and disallow that subdirectory.

Jim