Welcome to WebmasterWorld Guest from 54.145.209.34

Forum Moderators: goodroi

Banning a script using robots.txt

how do do this

   
9:47 pm on Apr 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi,

The URL's on my site are of the format
http*//domain.com/post.asp?id=****

I don't want search engines to crawl any URLs having the post.asp script. Would this work?

User-agent: *
Disallow: /post.asp

9:48 pm on Apr 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It should, if the spider reads and respects robots.txt -
9:50 pm on Apr 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thus, Google should follow it fine, right?
9:51 pm on Apr 20, 2004 (gmt 0)

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member



yes they should
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month