Welcome to WebmasterWorld Guest from 54.205.20.160

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Format

Will this work?

   
11:38 pm on May 5, 2005 (gmt 0)

10+ Year Member



User-agent: BecomeBot
Disallow: /
User-agent: Gigablast
Disallow: /
User-agent: www.agiftfor.co.uk
Disallow: /

I set out my robot.tct file as above to block these bots, the last one agiftfor is using a spider link validator that hits about 3k pages a day... my poor bandwidth is taking a beating.

11:47 pm on May 5, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




User-agent: Gigablast
Disallow: /

I am pretty sure Gigablast's bot is called Gigabot.

As for agiftfor you might want to make sure their bot has string that you specified in its useragent.

12:15 am on May 6, 2005 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



To be completely correct, you need a blank line after the last "Disallow" in each record, including the last record. And yes, Gigablast's robot is named Gigabot.

Jim

6:37 am on May 6, 2005 (gmt 0)

10+ Year Member



Cheers for the info.. all changed