Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt with wildcard?

         

tranceformer

2:32 pm on Jan 9, 2008 (gmt 0)

10+ Year Member



Googlebot is eating up about 30GB bandwidth per month.
I want to prevent him (and other robots) from spidering:

[mysite.com...]

and all derivatives thereof (e.g. [mysite.com...]

Is this possible with robots.txt? How would the code look?

Will this work?
User-agent: *
Disallow: /directory1/page1.php

or do I need to somehow specify a wildcard, e.g.
User-agent: *
Disallow: /directory1/page1.php?*
Thanks for your help!

coopster

4:39 am on Jan 11, 2008 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



Are you certain you want to Prevent content from appearing in Google search results [google.com]?

g1smd

12:39 am on Jan 13, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The partial URL in robots.txt is matched "from the left" so there is no point whatsoever in having wildcards at the extreme right of the disallow statement.