Welcome to WebmasterWorld Guest from 54.198.221.13

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Disallowed all bots by mistake - how to speed recovery?

     
10:18 pm on Oct 17, 2010 (gmt 0)

Full Member

10+ Year Member

joined:Jan 31, 2006
posts: 305
votes: 0


I've done something bad! Very bad!

I have an article database which lists several thousand articles.
I recently used that domain for some testing on another site i work on.
Because of this, i actually managed to change the WRONG robots.txt file, setting:

User-agent: *
Disallow: /

We all know what happens next.

It has been this way for several weeks, maybe 1.5 months!

I corrected it this evening - reason I found out was because all the adsense ads I have on the article directory only showed googles public ads.

Now, is there anything I can do to reduce the damage? People are still linking to me from all over the net - but will google remember that it is not allowed to crawl my site and hence not even check if it's "ok" again to come over to me?

Hoping for some fast replies :)
10:49 pm on Oct 17, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Google will check the robots.txt file regularly - I think you'll see that in just a couple days.

I would be curious as to how fast your rankings rebound. That is, whether googlebot will need to crawl every page again, or whether the URLs will just be restored for Google's back end (they definitely did not delete the data). So please keep us posted.

If you don't have an XML Sitemap, then I'd say create one and submit it through your Webmaster Tools account. And keep an eagle eye on WMT, too, so you notice anything that comes up as soon as possible.
4:08 am on Oct 18, 2010 (gmt 0)

Full Member

10+ Year Member

joined:Dec 27, 2006
posts:311
votes: 0


First think i would do is to submit a complete sitemap.
2:04 pm on Oct 18, 2010 (gmt 0)

Full Member

10+ Year Member

joined:Jan 31, 2006
posts: 305
votes: 0


Thank you both!

As a matter of fact, the site has been in the serps all the time, but not the articles that were approved by me after I denied the bot. But all the old articles have been in the index, and searchable.

Google actually started to show adsense ads just hours after I corrected the robots.txt.

I have not yet a complete sitemap, but I will make one right now.

Yes, I will pay attention to how it goes, and keep you posted.
I have posted a test-article to check how long it takes before it's indexed. Normally this shouldn't take more than a couple of days.
2:57 pm on Oct 18, 2010 (gmt 0)

Full Member

10+ Year Member

joined:Jan 31, 2006
posts: 305
votes: 0


By the way - I'm looking for a sitemap generator, but I can't find one that allows more than 300 - 1000 listings.

Any suggestions?
5:13 pm on Oct 18, 2010 (gmt 0)

Full Member

10+ Year Member

joined:Jan 31, 2006
posts: 305
votes: 0


Another update:

Everything goes smooth!
Google has already indexed my new article on my site which I published yesterday.

I did as you said - and made a sitemap which I uploaded to Google Webmastertools. It now seems like all the pages/categories/articles are back in business, and indexed by google.

All the adsense ads are back, so now I just wait for google also makes a cache of my article directory. (The cache:www.mydomain.com also disappeared when I denied the bots from visiting.)

So there you have it.

It does look like Google are catching up quick once allowed back into a site it has been indexing for several years - even though a webmaster screws up ;)
5:46 pm on Oct 18, 2010 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:7883
votes: 548


Google (and most SE bots) are pretty persistent. Happy you've been found "again". Then again, G and others keep hitting a site I "abandoned" back in 1997 so I am not surprised! :)
2:34 pm on Jan 4, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Apr 30, 2008
posts:2630
votes: 191


@I Will Make It

Could you update us on the progress of your recovery? Have you recovered fully and is your traffic / ranking on pre-mistake level?

Thanks
9:21 pm on Jan 22, 2011 (gmt 0)

Full Member

10+ Year Member

joined:Jan 31, 2006
posts: 305
votes: 0


aakk9999 - sorry my late reply. Didn't check my webmasterworld-email-account untill now.

But i'm happy to update you all, and very happy to say that everything is back to normal.

Again, I have top rankings, and lots of traffic coming my way.

After all - google does forgive mistakes - so it looks :)
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members