| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe and Support WebmasterWorld|
|Banning a script using robots.txt|
how do do this
| 9:47 pm on Apr 20, 2004 (gmt 0)|
The URL's on my site are of the format
I don't want search engines to crawl any URLs having the post.asp script. Would this work?
| 9:48 pm on Apr 20, 2004 (gmt 0)|
It should, if the spider reads and respects robots.txt -
| 9:50 pm on Apr 20, 2004 (gmt 0)|
Thus, Google should follow it fine, right?
| 9:51 pm on Apr 20, 2004 (gmt 0)|
yes they should
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved