Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Disallowed all bots by mistake - how to speed recovery?


I Will Make It

10:18 pm on Oct 17, 2010 (gmt 0)

5+ Year Member

I've done something bad! Very bad!

I have an article database which lists several thousand articles.
I recently used that domain for some testing on another site i work on.
Because of this, i actually managed to change the WRONG robots.txt file, setting:

User-agent: *
Disallow: /

We all know what happens next.

It has been this way for several weeks, maybe 1.5 months!

I corrected it this evening - reason I found out was because all the adsense ads I have on the article directory only showed googles public ads.

Now, is there anything I can do to reduce the damage? People are still linking to me from all over the net - but will google remember that it is not allowed to crawl my site and hence not even check if it's "ok" again to come over to me?

Hoping for some fast replies :)


10:49 pm on Oct 17, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Google will check the robots.txt file regularly - I think you'll see that in just a couple days.

I would be curious as to how fast your rankings rebound. That is, whether googlebot will need to crawl every page again, or whether the URLs will just be restored for Google's back end (they definitely did not delete the data). So please keep us posted.

If you don't have an XML Sitemap, then I'd say create one and submit it through your Webmaster Tools account. And keep an eagle eye on WMT, too, so you notice anything that comes up as soon as possible.


4:08 am on Oct 18, 2010 (gmt 0)

5+ Year Member

First think i would do is to submit a complete sitemap.

I Will Make It

2:04 pm on Oct 18, 2010 (gmt 0)

5+ Year Member

Thank you both!

As a matter of fact, the site has been in the serps all the time, but not the articles that were approved by me after I denied the bot. But all the old articles have been in the index, and searchable.

Google actually started to show adsense ads just hours after I corrected the robots.txt.

I have not yet a complete sitemap, but I will make one right now.

Yes, I will pay attention to how it goes, and keep you posted.
I have posted a test-article to check how long it takes before it's indexed. Normally this shouldn't take more than a couple of days.

I Will Make It

2:57 pm on Oct 18, 2010 (gmt 0)

5+ Year Member

By the way - I'm looking for a sitemap generator, but I can't find one that allows more than 300 - 1000 listings.

Any suggestions?

I Will Make It

5:13 pm on Oct 18, 2010 (gmt 0)

5+ Year Member

Another update:

Everything goes smooth!
Google has already indexed my new article on my site which I published yesterday.

I did as you said - and made a sitemap which I uploaded to Google Webmastertools. It now seems like all the pages/categories/articles are back in business, and indexed by google.

All the adsense ads are back, so now I just wait for google also makes a cache of my article directory. (The cache:www.mydomain.com also disappeared when I denied the bots from visiting.)

So there you have it.

It does look like Google are catching up quick once allowed back into a site it has been indexing for several years - even though a webmaster screws up ;)


5:46 pm on Oct 18, 2010 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

Google (and most SE bots) are pretty persistent. Happy you've been found "again". Then again, G and others keep hitting a site I "abandoned" back in 1997 so I am not surprised! :)


2:34 pm on Jan 4, 2011 (gmt 0)

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

@I Will Make It

Could you update us on the progress of your recovery? Have you recovered fully and is your traffic / ranking on pre-mistake level?


I Will Make It

9:21 pm on Jan 22, 2011 (gmt 0)

5+ Year Member

aakk9999 - sorry my late reply. Didn't check my webmasterworld-email-account untill now.

But i'm happy to update you all, and very happy to say that everything is back to normal.

Again, I have top rankings, and lots of traffic coming my way.

After all - google does forgive mistakes - so it looks :)

Featured Threads

Hot Threads This Week

Hot Threads This Month