Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt for yahoo and google.com

google and yahoo robots.txt



3:27 am on Jun 10, 2004 (gmt 0)

10+ Year Member

I don't want yahoo.com index whole website, and I don't want every spider index JAVA files.

Is this correct?

User-agent: Slurp
Disallow: /

User-agent: *
Disallow: /*.js$


Many Thank

Abdelrhman Fahmy

4:18 am on Jun 10, 2004 (gmt 0)

10+ Year Member

Correct, and you may look at

finally some one who don't want Yahoo to index his website! :)


4:47 am on Jun 10, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Only Google will recognize the "wildcard" *.js -- It is not standard syntax.

The standard robots.txt uses prefix matching, which means that you will have to disallow each .js file individually, or place them all in a subdirectory and disallow that subdirectory.



Featured Threads

Hot Threads This Week

Hot Threads This Month