| Banning a script using robots.txt
how do do this |
Imaster
msg:1526184 | 9:47 pm on Apr 20, 2004 (gmt 0) |
Hi, The URL's on my site are of the format
http*//domain.com/post.asp?id=**** I don't want search engines to crawl any URLs having the post.asp script. Would this work? User-agent: *
Disallow: /post.asp
|
bcolflesh
msg:1526185 | 9:48 pm on Apr 20, 2004 (gmt 0) |
It should, if the spider reads and respects robots.txt -
|
Imaster
msg:1526186 | 9:50 pm on Apr 20, 2004 (gmt 0) |
Thus, Google should follow it fine, right?
|
jatar_k
msg:1526187 | 9:51 pm on Apr 20, 2004 (gmt 0) |
yes they should
|
|
|
