Forum Moderators: goodroi

Message Too Old, No Replies

Specifically allow one bot to index a page

please check my code ;)

         

RonPK

3:48 pm on Mar 26, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I need to ban all bots but the AdSense bot from a calendar page. Why? Because every page contains a link to the next month. Links look like calendar.php?month=4&year=2005. Bots could go on crawling until the end of time - literally.

Here's the complete robots.txt:

User-agent: Mediapartners-Google* 
Disallow:

User-agent: *
Disallow: calendar.php

Am I right that the lines for the Adsense-bot should come before the rest? Or should it be the other way around?

jimbeetle

4:43 pm on Mar 26, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Looks fine, just like the example to allow a single bot at robotstxt.org [robotstxt.org].

DanA

5:42 pm on Mar 26, 2005 (gmt 0)

10+ Year Member



I think that
Disallow: /calendar.php
would be better

jimbeetle

5:57 pm on Mar 26, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Ooops! And I looked at that at least six times.

RonPK

10:37 am on Mar 27, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



OK, thanks for the swift replies.