Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Format

Will this work?



11:38 pm on May 5, 2005 (gmt 0)

10+ Year Member

User-agent: BecomeBot
Disallow: /
User-agent: Gigablast
Disallow: /
User-agent: www.agiftfor.co.uk
Disallow: /

I set out my robot.tct file as above to block these bots, the last one agiftfor is using a spider link validator that hits about 3k pages a day... my poor bandwidth is taking a beating.

Lord Majestic

11:47 pm on May 5, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

User-agent: Gigablast
Disallow: /

I am pretty sure Gigablast's bot is called Gigabot.

As for agiftfor you might want to make sure their bot has string that you specified in its useragent.


12:15 am on May 6, 2005 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

To be completely correct, you need a blank line after the last "Disallow" in each record, including the last record. And yes, Gigablast's robot is named Gigabot.



6:37 am on May 6, 2005 (gmt 0)

10+ Year Member

Cheers for the info.. all changed

Featured Threads

Hot Threads This Week

Hot Threads This Month