Welcome to WebmasterWorld Guest from 54.144.68.27

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Format

Will this work?

     

kbba04527

11:38 pm on May 5, 2005 (gmt 0)

10+ Year Member



User-agent: BecomeBot
Disallow: /
User-agent: Gigablast
Disallow: /
User-agent: www.agiftfor.co.uk
Disallow: /

I set out my robot.tct file as above to block these bots, the last one agiftfor is using a spider link validator that hits about 3k pages a day... my poor bandwidth is taking a beating.

Lord Majestic

11:47 pm on May 5, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




User-agent: Gigablast
Disallow: /

I am pretty sure Gigablast's bot is called Gigabot.

As for agiftfor you might want to make sure their bot has string that you specified in its useragent.

jdMorgan

12:15 am on May 6, 2005 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



To be completely correct, you need a blank line after the last "Disallow" in each record, including the last record. And yes, Gigablast's robot is named Gigabot.

Jim

kbba04527

6:37 am on May 6, 2005 (gmt 0)

10+ Year Member



Cheers for the info.. all changed
 

Featured Threads

Hot Threads This Week

Hot Threads This Month