homepage Welcome to WebmasterWorld Guest from 204.236.254.124
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt Format
Will this work?
kbba04527




msg:1529468
 11:38 pm on May 5, 2005 (gmt 0)

User-agent: BecomeBot
Disallow: /
User-agent: Gigablast
Disallow: /
User-agent: www.agiftfor.co.uk
Disallow: /

I set out my robot.tct file as above to block these bots, the last one agiftfor is using a spider link validator that hits about 3k pages a day... my poor bandwidth is taking a beating.

 

Lord Majestic




msg:1529469
 11:47 pm on May 5, 2005 (gmt 0)


User-agent: Gigablast
Disallow: /

I am pretty sure Gigablast's bot is called Gigabot.

As for agiftfor you might want to make sure their bot has string that you specified in its useragent.

jdMorgan




msg:1529470
 12:15 am on May 6, 2005 (gmt 0)

To be completely correct, you need a blank line after the last "Disallow" in each record, including the last record. And yes, Gigablast's robot is named Gigabot.

Jim

kbba04527




msg:1529471
 6:37 am on May 6, 2005 (gmt 0)

Cheers for the info.. all changed

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved