homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Disallowed all bots by mistake - how to speed recovery?
I Will Make It

5+ Year Member

Msg#: 4218063 posted 10:18 pm on Oct 17, 2010 (gmt 0)

I've done something bad! Very bad!

I have an article database which lists several thousand articles.
I recently used that domain for some testing on another site i work on.
Because of this, i actually managed to change the WRONG robots.txt file, setting:

User-agent: *
Disallow: /

We all know what happens next.

It has been this way for several weeks, maybe 1.5 months!

I corrected it this evening - reason I found out was because all the adsense ads I have on the article directory only showed googles public ads.

Now, is there anything I can do to reduce the damage? People are still linking to me from all over the net - but will google remember that it is not allowed to crawl my site and hence not even check if it's "ok" again to come over to me?

Hoping for some fast replies :)



WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 4218063 posted 10:49 pm on Oct 17, 2010 (gmt 0)

Google will check the robots.txt file regularly - I think you'll see that in just a couple days.

I would be curious as to how fast your rankings rebound. That is, whether googlebot will need to crawl every page again, or whether the URLs will just be restored for Google's back end (they definitely did not delete the data). So please keep us posted.

If you don't have an XML Sitemap, then I'd say create one and submit it through your Webmaster Tools account. And keep an eagle eye on WMT, too, so you notice anything that comes up as soon as possible.


5+ Year Member

Msg#: 4218063 posted 4:08 am on Oct 18, 2010 (gmt 0)

First think i would do is to submit a complete sitemap.

I Will Make It

5+ Year Member

Msg#: 4218063 posted 2:04 pm on Oct 18, 2010 (gmt 0)

Thank you both!

As a matter of fact, the site has been in the serps all the time, but not the articles that were approved by me after I denied the bot. But all the old articles have been in the index, and searchable.

Google actually started to show adsense ads just hours after I corrected the robots.txt.

I have not yet a complete sitemap, but I will make one right now.

Yes, I will pay attention to how it goes, and keep you posted.
I have posted a test-article to check how long it takes before it's indexed. Normally this shouldn't take more than a couple of days.

I Will Make It

5+ Year Member

Msg#: 4218063 posted 2:57 pm on Oct 18, 2010 (gmt 0)

By the way - I'm looking for a sitemap generator, but I can't find one that allows more than 300 - 1000 listings.

Any suggestions?

I Will Make It

5+ Year Member

Msg#: 4218063 posted 5:13 pm on Oct 18, 2010 (gmt 0)

Another update:

Everything goes smooth!
Google has already indexed my new article on my site which I published yesterday.

I did as you said - and made a sitemap which I uploaded to Google Webmastertools. It now seems like all the pages/categories/articles are back in business, and indexed by google.

All the adsense ads are back, so now I just wait for google also makes a cache of my article directory. (The cache:www.mydomain.com also disappeared when I denied the bots from visiting.)

So there you have it.

It does look like Google are catching up quick once allowed back into a site it has been indexing for several years - even though a webmaster screws up ;)


WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

Msg#: 4218063 posted 5:46 pm on Oct 18, 2010 (gmt 0)

Google (and most SE bots) are pretty persistent. Happy you've been found "again". Then again, G and others keep hitting a site I "abandoned" back in 1997 so I am not surprised! :)


WebmasterWorld Administrator 5+ Year Member

Msg#: 4218063 posted 2:34 pm on Jan 4, 2011 (gmt 0)

@I Will Make It

Could you update us on the progress of your recovery? Have you recovered fully and is your traffic / ranking on pre-mistake level?


I Will Make It

5+ Year Member

Msg#: 4218063 posted 9:21 pm on Jan 22, 2011 (gmt 0)

aakk9999 - sorry my late reply. Didn't check my webmasterworld-email-account untill now.

But i'm happy to update you all, and very happy to say that everything is back to normal.

Again, I have top rankings, and lots of traffic coming my way.

After all - google does forgive mistakes - so it looks :)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved