Welcome to WebmasterWorld Guest from 54.161.88.189

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt for yahoo and google.com

google and yahoo robots.txt

     
3:27 am on Jun 10, 2004 (gmt 0)

New User

10+ Year Member

joined:May 2, 2003
posts:16
votes: 0


I don't want yahoo.com index whole website, and I don't want every spider index JAVA files.

Is this correct?
----------------------------------------------------------------

User-agent: Slurp
Disallow: /

User-agent: *
Disallow: /*.js$

-----------------------------------------------------------------

Many Thank

4:18 am on June 10, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 7, 2003
posts:120
votes: 0


Correct, and you may look at
[help.yahoo.com...]

finally some one who don't want Yahoo to index his website! :)

4:47 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Only Google will recognize the "wildcard" *.js -- It is not standard syntax.

The standard robots.txt uses prefix matching, which means that you will have to disallow each .js file individually, or place them all in a subdirectory and disallow that subdirectory.

Jim