Forum Moderators: open

Message Too Old, No Replies

Will this work

Trying to get my robots.txt file right

         

le_gber

11:38 am on Aug 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi,

After having problems with Googlebot on one of my domain names I want to keep her away from on domain and only allow her to another.

Could you tell me if the following is the right way to do it.

On the domain I DON'T want GBot
User-agent: googlebot
Disallow: /

User-agent: *
Disallow: /cgi-bin/

On the domain I DO want Gdot
User-agent: googlebot
Disallow: /cgi-bin/

User-agent: *
Disallow: /

Is that right?

Leo

killroy

11:45 am on Aug 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Why not:

On the domain I DON'T want GBot
User-agent: googlebot
Disallow: /

On the domain I DO want Gdot
User-agent: googlebot
Disallow:

SN

le_gber

2:28 pm on Aug 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



because there is other folder that I don't want all the bot to crawl

Leo