Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

7 year old site totally removed. No evidence of problems. Advice?

         

drall

1:18 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We have had a 7 year old site of ours totally removed from the index.

Site: command shows no urls. The site is a pretty respected site at pr 6 and has loads of great links.

Webmaster tools still shows sitelinks and backlink data and my security guy cant see any hacks or exploits.

Any suggestions?

Should I file a reconsideration request? I dont even know what we did wrong, we havent done anything differently on the site that we havent been doing for 7 years.

tedster

1:44 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well, it's one of two basic things - something wrong at Google or something wrong for your site - either the server, or on-page or offpage. How long ago did the site: operator go to zero for you?

I assume you are seeing no google search traffic, correct?

drall

1:50 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for the help Tedster, I noticed no traffic from Google today for the site. Checked G and bammo totally gone. Traffic just totally stopped yesterday at some point so I am guessing that is when the site was removed.

Log files do not show any problems, changed my user agent and site is serving properly. Site has been online and no problems with dns or the server for the last week that I looked at. Security seems intact, its just plain html and the server is really locked down.

drall

2:00 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Is it normal for the link and related site command to still be functional after a removal?

tedster

2:03 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, these secondary operators often show a time delay.

If you're pretty certain there are no technical issues, you might submit a reconsideration request. You also might take up the issue in Google's own webmaster help forum.

But check some basics first - is there a robots.txt file on the site and does it allow googlebot to crawl? More than one robots.txt accident has occurred to webmasters here.

Similarly, make sure that somehow robots meta tags did not get changed to noindex.

It takes some pretty severe spamming for Google to outright ban a site from their index these days. From what you described, I doubt that's the case for you. So assume there is something that can be found and fixed either by Google or by your team - and best wishes for a speedy recovery.

tedster

2:05 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would also check your recent server logs to see how googlebot has been treated.

drall

2:06 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Tedster, we will pour over the technical stuff some more.

drall

2:27 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Googlebot comes back ok in the logs, G is still crawling hundreds of pages as we speak. Does googlebot usually crawl sites after removal?

tedster

2:32 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, googlebot will request URLs from even a banned site once in a while - but usually nothing like the volume you're talking about. Is googlebot getting mostly 200 and 304 responses?

drall

2:37 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yep 200 and 304 to all requests. I have 2 guys here pouring over stuff, we just cant see anything wrong yet. Have people had sites disappear temp and then pop back? Maybe like some glitch at Google?

tedster

2:56 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It's rare but it does happen. I would suggest a reconsideration request. They always get read and that should be enough to get a mere glitch fixed.

drall

3:21 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Ok thanks Tedster, I will file one right now.

kidder

4:35 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



drall - I've been through this sort of thing recently and we found some links that had been forced into a blog post, we got it sorted working with Google's advice in less than a week(webmaster tools). If your clean and don't have any sins then Google will be working with you.

drall

5:00 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Kidder. To the best of my knowledge we are clean. The site really hasnt changed in over half a decade design wise and its all static html.

We have been featured on forbes and fox finance and tons of other sites. I cant find a technical reason yet at least on our end so I hope they can help us figure out what the heck is going wrong.

johnnie

11:39 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A complete, rapid deindexation smells like malware or another hack of some sorts. For starters, try [google.com...]

g1smd

12:00 pm on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Last week, a client deployed a complete CMS redesign (without telling me that they were going live with it) with 'friendly' URLs and such, and a week later I was called in to evaluate the changeover.

Google had already severely reduced the site listings... as every domain now had a site-wide disallow in the robots.txt file. They'd also cleared their analytics account number from the Tools section, left the default (example@example.com) contact email address for their contact form in the Comms settings, and removed all of the .htaccess rules that redirected from old CMS URLs to new CMS URLs (and introduced a code typo that stopped every page showing any meta description).

Google will take a few months to get things back where they were...

[edited by: g1smd at 12:19 pm (utc) on June 5, 2009]

drall

12:00 pm on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi Johnnie, site came back clean for that. Security says rootkit comes back clean. Spoofed googlebot agent and site loads properly.

This is really getting me down, clean little site that due to content, age and links was competing with fortune 100 companies for great terms for years then one morning you wake up and poofta gone.

I actually showed this site to my wife as a example of how Google lets small sites compete with gorillas for big terms as long as you have good content.

[edited by: tedster at 8:10 pm (utc) on June 5, 2009]

drall

3:29 pm on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Using [google.com...] the site comes back clean.

But it says google hasnt visited the site in the last 90 days but we had traffic up to 2 days ago and googlebot is still activly crawling pages. What the heck?

g1smd

3:50 pm on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Did at any time your site return a 404 status for valid URLs and someone use the Google URL Removal Tools on your site while that was happening?

drall

4:39 pm on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just checked the removal section of wmt, I dont see any removal requests. As far as I know the site hasnt been returning 404's but I will look over the last few weeks of logs and see if I can find any instances.

I submitted a reconsideration request last night. Hope they can shed light as to whats going on.

The only thing we have done differently with the site vs the last 7 years is test out microsofts ads on it.

chrism

6:07 pm on Jun 5, 2009 (gmt 0)

10+ Year Member



Hi drall

We're seeing something similar at the moment - previously 5000+ indexed pages disappearing what appears to be instantly on may 28th. Googlebot is also going nuts on our site - if anything more crawling than before.

site: query returns a full list as normal.

We did have a lot of internal links pointing at subfolders which redirected to subfolder/ by 301 which we only spotted by going through the logs after this happened, and googlebot appears to be respidering the site bit by bit at the moment. We have corrected the links and hopefully will reappear soon fingers crossed.

An interesting observation is we appear in the serps as normal for phrases if we put speech marks around them - in other words where we used to rank for key phrase we now don't appear at all, but for "key phrase" we appear as before. I've not seen this behaviour before.