| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|Banning a script using robots.txt|
how do do this
| 9:47 pm on Apr 20, 2004 (gmt 0)|
The URL's on my site are of the format
I don't want search engines to crawl any URLs having the post.asp script. Would this work?
| 9:48 pm on Apr 20, 2004 (gmt 0)|
It should, if the spider reads and respects robots.txt -
| 9:50 pm on Apr 20, 2004 (gmt 0)|
Thus, Google should follow it fine, right?
| 9:51 pm on Apr 20, 2004 (gmt 0)|
yes they should
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2014 all rights reserved