homepage Welcome to WebmasterWorld Guest from 54.211.181.45
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Changing your Mind with robots.txt
Will googlebot come back to the party?
Spine

10+ Year Member



 
Msg#: 499 posted 5:53 am on Nov 26, 2004 (gmt 0)

A couple of weeks ago I banned googlebot from a underused domain using robots.txt , now I'm thinking I want to change the content on that domain and it's pages, and let G-bot back in.

I think G-bot has come around 3-4 times and found itself excluded with robots.txt. If I remove the r line referring to gbot in my robots.txt, will it come back at some point, or does it only take the 'insult' so many times before it gives up? As I said, it's only been about 14 days.

I looked around and didn't find any info on this.

Cheers

 

Spine

10+ Year Member



 
Msg#: 499 posted 4:41 pm on Nov 26, 2004 (gmt 0)

Update:

The bot came back today, about 4 days after it's last visit (when it wasn't welcome).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved