homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How long before bots re-visit after they were told they are not allowed?

Msg#: 3575470 posted 5:26 am on Feb 15, 2008 (gmt 0)

I was dumb enough to block all robots but the biggest search engines. I saw Gigablast's robot and others were on a single-hit visit (most likely robots.txt), saw that they are not welcome and then went away. :(

Now I have changed my robots.txt into allowing all bots, instead blocking bad User-agents by force manually. Will Gigabot and others ever come back? If so, how long before they come back and see that they are now allowed in?

I feel so stupid for losing the traffic.



WebmasterWorld Senior Member quadrille us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 3575470 posted 10:53 am on Feb 15, 2008 (gmt 0)

If you have links to your site, and good site navigation, they'll be back.

Just a matter of time, and that depends on all manner of unknowables.

But they'll be back - that's their job!


5+ Year Member

Msg#: 3575470 posted 3:32 am on Mar 17, 2008 (gmt 0)

How long after the robots visit before the search engines include your site? I can see that several robots have visited in the past 24 hrs, but I am not listed on their search engines, I was visited by googlebot, yahoo slurp, msn media and a couple of unidentified ones....how long do I need to wait before resubmitting after changing?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved