| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
Will this work?
| 11:38 pm on May 5, 2005 (gmt 0)|
I set out my robot.tct file as above to block these bots, the last one agiftfor is using a spider link validator that hits about 3k pages a day... my poor bandwidth is taking a beating.
| 11:47 pm on May 5, 2005 (gmt 0)|
I am pretty sure Gigablast's bot is called Gigabot.
As for agiftfor you might want to make sure their bot has string that you specified in its useragent.
| 12:15 am on May 6, 2005 (gmt 0)|
To be completely correct, you need a blank line after the last "Disallow" in each record, including the last record. And yes, Gigablast's robot is named Gigabot.
| 6:37 am on May 6, 2005 (gmt 0)|
Cheers for the info.. all changed
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved