| Robots.txt for yahoo and google.com
google and yahoo robots.txt |
adisakmai
msg:1526877 | 3:27 am on Jun 10, 2004 (gmt 0) |
I don't want yahoo.com index whole website, and I don't want every spider index JAVA files. Is this correct?
---------------------------------------------------------------- User-agent: Slurp
Disallow: / User-agent: *
Disallow: /*.js$ ----------------------------------------------------------------- Many Thank
|
Abdelrhman Fahmy
msg:1526878 | 4:18 am on Jun 10, 2004 (gmt 0) |
Correct, and you may look at
[help.yahoo.com...] finally some one who don't want Yahoo to index his website! :)
|
jdMorgan
msg:1526879 | 4:47 am on Jun 10, 2004 (gmt 0) |
Only Google will recognize the "wildcard" *.js -- It is not standard syntax. The standard robots.txt uses prefix matching, which means that you will have to disallow each .js file individually, or place them all in a subdirectory and disallow that subdirectory. Jim
|
|
|
