Welcome to WebmasterWorld Guest from 54.196.244.186

Forum Moderators: goodroi

Message Too Old, No Replies

Banning a script using robots.txt

how do do this

     
9:47 pm on Apr 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


Hi,

The URL's on my site are of the format
http*//domain.com/post.asp?id=****

I don't want search engines to crawl any URLs having the post.asp script. Would this work?

User-agent: *
Disallow: /post.asp

9:48 pm on Apr 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


It should, if the spider reads and respects robots.txt -
9:50 pm on Apr 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


Thus, Google should follow it fine, right?
9:51 pm on Apr 20, 2004 (gmt 0)

Administrator

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 24, 2001
posts:15755
votes: 0


yes they should