Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt for yahoo and google.com

google and yahoo robots.txt

3:27 am on Jun 10, 2004 (gmt 0)

New User

10+ Year Member

joined:May 2, 2003
votes: 0

I don't want yahoo.com index whole website, and I don't want every spider index JAVA files.

Is this correct?

User-agent: Slurp
Disallow: /

User-agent: *
Disallow: /*.js$


Many Thank

4:18 am on June 10, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 7, 2003
votes: 0

Correct, and you may look at

finally some one who don't want Yahoo to index his website! :)

4:47 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

Only Google will recognize the "wildcard" *.js -- It is not standard syntax.

The standard robots.txt uses prefix matching, which means that you will have to disallow each .js file individually, or place them all in a subdirectory and disallow that subdirectory.