Forum Moderators: open

Message Too Old, No Replies

Getting back into Google after robots.txt mistake

What length of time am I looking at

         

pcgamez

12:02 am on Aug 10, 2004 (gmt 0)

10+ Year Member



A few weeks ago, I accidently blocked all spiders from every page of my site for a period of about 2-3 weeks. I have since corrected that error. Yahoo Slurp, and the Google MediaPartners (AdSense) spider both started crawling my site again, but the main Googlebot has yet to after about 4 days. Does anyone have any experience with this?

btw, site is PR6 and used to get about 600-900 visits/day from Google, and about 20k/month from Googlebot.

uncle_bob

11:50 am on Aug 10, 2004 (gmt 0)

10+ Year Member



I had exactly the same problem a few months ago, took about 3 to 4 weeks before googlebot visits and serps started returning to normal.

PhoneGuy

2:09 am on Aug 11, 2004 (gmt 0)

10+ Year Member



Same here. The SysAdmin accidently copied a bad robots.txt into the root and blocked everything. We didn't notice it at first. After about a week+ we realized what had happened. It took about 4 - 6 weeks to start seeing our page count going up again. Even after removing the robots.txt file our page count continued to go down. I guess it was just Google's backlog of work. What a nightmare that was.

pcgamez

3:01 am on Aug 11, 2004 (gmt 0)

10+ Year Member



I would never have suspected anything was wrong if the number of visits has not dropped to 1/40th their normal. I guess that means that sites that block Googlebot are penalized in some fashion.