Site disappeared from Google around December 10, 2007

[gomer]

A site of mine site 'mostly' disappered from Google around the time shown in the title and site: command is displaying odd behaviour. I am posting this to find out if others have experinced something similar.

I should explain 'mostly' above. My site had thousands of pages indexed.

Now when I do site: Google does show an accurate page count (in the thousands which it used to have) but when I try and scroll through the pages, Google only shows me about 150 page are in the index.

These lost pages are not in the index (despite the accurate site: ) because when I search for snippets that are unique to those pages, Google returns no results.

My homepage is still in the index along with those 150 pages. My homepage still ranks for a very competitive term. It is second for a query with 240K results (last week this same query showed 2.4 million results).

My referrals from Google are now less than 1% of what they were prior to 7pm EST.

I am hoping this is an issue of Google temporarily losing a site.

I really don't think this is a penalty issue, just a lost site issue. Would like to know if anyone else is experiencing the same thing.

Thanks
gomer

[jgold454]

As stated in my other post, my site completely vanished around the same time. I have had some DNS issues due to a server move.

[Vimes]

Now when I do site: Google does show an accurate page count (in the thousands which it used to have) but when I try and scroll through the pages, Google only shows me about 150 page are in the index.

"In order to show you the most relevant results, we have omitted some entries very similar"

Are you receiving the omitted link at the bottom of those 150 pages?

Vimes.

[gomer]

Are you receiving the omitted link at the bottom of those 150 pages?

No Vimes, I am not.

[Vimes]

Are all DC's showing you these results or is it just the DC's, i'm having the exact opposite happen to my site where Google has inflated my page totals by almost half.

whats the page totals for your site on these DC's

72.14.207.107
64.233.189.107

the first for me shows correct figures the second is bloated.

Vimes.

[TigerSBT]

Don't know exactly when but it must have been roughly about the same time that 4 of our sites disappeared from the index. Only the link: command returns any information from google.

These 4 sites are all listed in my webmaster account and still show pagerank, if that is any consolation.

I will investigate more and report back if I find anything.

[Atomic]

I noticed something similar for several of my sites. One is a very large site with 10's of thousands of pages. Everything was back to normal within a few hours. As I said, this wasn't one site. It was several, unrelated sites. All returned to normal. Hang tight.

[gomer]

TigerSBT, Atomic, thanks for sharing that.

Atomic, did you lose the site you mentioned around the same time? when you lost it, did all traffic vanish? It is now 6am EST on December 11th, no return of the tens of thousands of pages.

Vimes, it is all DC's not just the ones you mentioned in your post above.

[Atomic]

No, all of my traffic did not vanish. I have far too many IBLs for that to happen. You posted 7 hours after you initial poast indicating your site had not come back. Is it still missing?

[gomer]

Atomic, the site is not back yet. Our site did not completely vanish either. There are about 150 pages left (from tens of thousands), we have strong inbound links as well.

We're still second on a search term that has 240K results (which used to have 2.4 million).

site:example.com still shows the correct page count, except they are not in the index.

site:example.com/* shows the correct number in the main index as well (non-supplemental) but these pages are not there.

Have been reading at other forums and this has happened to other sites as well.

[jgold454]

when I do an inurl:mysite.com I am getting a proxy hijacked site that is appearing...My site is still nowhere to be found but if I do a link:mysite.com it is showing some of my backlinks..

[superbitz]

To those who are interested, I lost 3 sites last night 12/10 at around 7pm. All 3 sites were top tier sites with 1000's of backlinks and which had been on the front page of Google for their major keywords. The sites are gone. All pages gone, and no cache date returned.

My clients are freaking out but there isn't much I can do. We have been very white hat and this has really thrown us for a loop. I see no reason for the sites to have beeen dropped completely. I hope this is a glitch and the sites "pop back in" in the next 48 hours. If not, I guess I better brace for that "Google coal" in my stocking this holiday season.

[divine7]

My site with more than 1000 pages indexed, and 20k+ backlinks got dropped out as well, de-indexed completely last night.

I did nothing fishy with the site, I followed all WH approach. A number of times my site also hit the homepage of digg.

I don't know whats going on with Google here. Though the PR is as it is, so I am hoping its a glitch, not a ban.

[divine7]

I just noticed a couple proxy sites duplicating my entire site as it is, and they are in the google index.

Maybe the site dropped out from the index for duplicate content via a proxy hijack?

Anyone like to share his/her view?

[gomer]

divine7, the proxy hijack theory has surfaced on other forums. For my site, I found only one page that was proxy hijacked. I found the page by doing the following at Google:

inurl:example.com proxy

If I am missing something on how to find proxy hijacks, please let me know. If I got it right, since I have only one page hijacked, I am thinking that is not the issue for me.

[superbitz]

I found a proxy hijack on one of my "disappeared" sites just this evening.

This is a first for me. I have taken the following steps and would suggest those of you who have suffered this problem do the same:

1. Add this to all of your headers:
<base href="http://www.yoursite.com/" />

and if you see an attempted hijack...

2. Block the site via .htaccess:
RewriteCond %{HTTP_REFERER} yourproblemproxy\.com

3. Block the IP address of the proxy
order allow,deny
deny from 11.22.33.44
allow from all

4. Do your research and file a spam report with Google.
[google.com...]

[edited by: tedster at 5:41 am (utc) on Dec. 12, 2007]
[edit reason] removed specifics [/edit]

[gomer]

Thanks superbitz but I think your post may get edited.

Question: how can I do a query to find out if my site has been hijacked by a proxy. I don't think the query I have in my post above will work for all instances.

[tedster]

Defending against proxy hijacks can be a challenge. We have a good discussion about this in our Hot Topics area [webmasterworld.com], which is always pinned to the top of this forum's index page:

Proxy Server URLs Can Hijack Your Google Ranking - how to defend? [webmasterworld.com]

[gomer]

Thanks Tedster, I looked through those proxy hijack threads and there is a lot in there.

Tedster or anyone, is there a query I can do to tell if my pages have been hit by a proxy hijack?

I tried:

inurl:example.com -site:example.com

Can I just look through those results and look for proxy jacks like that?

If I am just seeing one page hijacked, do I have anything to worry about? We will block that domain.

I don't feel this is the cause of my site dropping out of the index but I do need to look into this fully to make sure.

Thanks.

[superbitz]

Thanks tedster -- this is a new situation for me. I value the link and the information on this issue and will implement the steps first thing in the morning.

[tedster]

gomer, that is a good search to see if Google has indexed any proxy urls that go to your server. However, if your rankings have been hijacked by a proxy url, then the proxy url will be sitting there in the search results, right where your domain's url used to be - that's what makes it a hijack.

[TigerSBT]

The pages of my sites have been removed from google, not one left. If it were a proxy hijack then I would expect the site: command to show listings of pages with the proxy url.

Also, if I search for my domain name then there are no results found, unlike the proxy hijacks

Commands that still work are
link: inurl: and related:
everything else returns 0 results.

Is there any other way of checking for proxy hijacks?

I have a idea that this is related to advertising that we have took part in within the last 6 months.

[navonna]

TigerSBT - would that advertising be paid blog posts?

[Stiphen]

So I have been hit too and from what I can tell this seems to be a big update. If you are looking at the search competition before this happened.

"keyword-1" before 740,000,000 now 77,000,000
"keyword-2" 24,300,00 now 172,000
"keyword-3" 86,600,000 now 12,200,000
"keyword-4" 107,000,000 now 12,000,000

I could continue all day with this and it would show the same thing, 90% of the supposedly indexed pages are gone.

One of our sites lost a bit more than 90% but we are still being crawled at our normal rate which is 2000-3000 pages a day. Our domain still ranks for pretty competitive terms and pass weight to others and those pages we still have in the index.

#*$! is going on?

[edited by: tedster at 5:38 pm (utc) on Dec. 12, 2007]
[edit reason] no specific searches, please [/edit]

[gomer]

Tedster, thanks for the response. This make it very clear on the proxy hijack. This is not what is happening to me.

Stiphen, yes, the query counts are also down sharly but this seems to have been happening even a few weeks before the sites disappeared.

Unlike some others on here, site: still works for my site, the pages are just not in the index.

[jgold454]

Would a banned site still be showing up in google maps results? I am still seeing some traffic from there yet my site is nowhere to be found in the regular index.

[gomer]

jgold454, I really don't think this is a ban. The pages of my site that are in the index do rank well, I just have very few pages left.

jgold454, when you do site:, do you get any pages shown.

[jgold454]

I get no pages in my site when i do:

site:mysite.com
url:mysite.com

when I do inurl:www.mysite.com I get a proxy site.

When I do link:mysite.com it shows my backlinks.

Yet I still have pagerank across all the datacenters. In my webmaster account it says:

No pages from your site are currently included in Google's index. Indexing can take time. You may find it helpful to review our information for webmasters and webmaster guidelines.

[gomer]

jgold454, I also have pagerank for pages that are not in Google's index.

OUr plan is to wait a while and just see if this situation resolves itself.

For sites that I know are highly penalized or very weak, I still see their pages in the index, they just don't rank for anything.

Since I am not cloaking, using hidden text etc, I expect these pages to at least come back in the index at some point.

[tedster]

Hm - rethinking this a bit, most especially jgold454's case, it does sound like the proxy urls may be a problem, even if they are not ranking where the original domain used to rank. I would at least do a spam report - but that's made more difficult for you if there's no regular search that shows the proxy urls.

I would say check googlebot activity in your server logs, see if there is any. A proxy url becomes a problem when the real googlebot indexes your content through a proxy server and uses their IP address. This is why doing the reverse/forward DNS check defends against the problem.