Index page is vanishing temporarily

[doc_z]

One of my sites, which was affected by the 'index page dropped phenomenon' (bug, penalty or whatever), is vanishing temporarily from the results (when searching for keyword1). The problem often occurs when searching the first time. When the result page is reloaded, normally the page reappears.

The problem doesn't seem to be related to a specific data centre (i.e. it can occur at all of them). The problem also arises for a search of allinanchor, allintitle and allintext of keyword1. However, it seems that other keywords or combination of keywords are not affected.

Anybody else seeing a similar behaviour?

[drewls]

Yes, I changed the page title slightly. I should correct myself. I am now not only experiencing the refresh problem, but also the old lost index page phenom of yore.

The refresh thing is still going on, but the other seems a bit more stable now. I may have been a bit quick to judge that one.

I'd probably be less paranoid about the whole thing if we could get some kind of word that this is being worked on. (hint hint)

[mipapage]

Hey all,

Hoping to get a summary of this thread here, and then maybe point that summary in a direction... (every time I see a new post here I think it's GG with his input...)

---

Problem...

doc_z's original post outlined one of the problems, while the other has been around a bit:

1. Search for Keyword A and the index page of your site isn't there. Hit refresh and it's back. Doesn't happen this way for all of your keywords.

2. The index page isn't there. End of point. Don't even bother refreshing.

Explanations?

Now, Napoleon had a good look at problem #2 after Esmerelda (hmm, can't find the link..), but problem #1 is a bit different.

I can't figure it out from the sites we have worked on. It's not age, keyword density or inbound anchor text, from what I can tell.

Questions for us all to maybe help GoogleGuy see a pattern:

1. Did everyone here have both problems with their site?
   
2. Have you switched hosts lately? Are you virtual?
   
3. I've seen a lot of 'refresh and it's #1'. This is the case for me, how about the rest of you?
   
4. Are all of these sites in the Directory?
   
5. When was your site first into Google/Directory?
   
6. Were you the first person to own your domain (we weren't!)

---

Me:

1. Yes
   
2. Yes/Yes
   
3. Yes
   
4. Yes
   
5. December/January
   
6. No
   

[Brian]

I hate to disappoint those who maybe think this problem can be traced to some recent changes. I reported this vanishing index/freshbot/hit-the-button-again phenomenon at least six months ago - on this board and others. I made repeated posts asking if anyone could explain it - and nobody could. Eventually I gave up asking.

I deduce from the deluge here all these months later that some kind of bug is cascading through Google's system, worsening over time.

I'm not surprised to see GoogleGuy expressing skepticism that this problem even exists. When I queried GoogleGuy's identity and role on this board my posts were deleted.

[Noticing the date I joined this forum, I may have first noted the problem eight or nine months ago.]

I don't believe that all is as it appears to be.

[WebFusion]

The "refresh" problem has never applied in my case. My index page, although still technically "indexed", still fails to appear (for the first time in about 2 1/2 years) for primary keyword/phrase.

Site is kept relatively fresh, with approx. 50-75 new pages of fresh content added per month, as well as weekly/bi weekly updates to main page (nothing major, as design has lasted in top 5 for 2+ years).

Have checked all link partners, and none seem to be "bad neighborhoods", although, since we've maintained our PR6 (although I'm not sure how reliable that is anymore), I don't think we've incurred any sort of penalty per se.

Since I've seen this occurring to other sites in my general vicinity in the SERPs (the latest occurred just last week, for a LONG TIME competiotr with a much larger/older site), with a return to "normal" listings within a week or so, I tend to think the "New" google rolling update thingy is the culprit behind all this relative inconsistency/instability.

In retrospect, this is the first time in almost 3 years that any changes at google have affected our site. Even with all the algo-changes in the last year or so, we scarcely moved a spot or two.

Having said all that, and looking at the state of the current results, it looks to me to be a beta-test on a grand scale, one which is causing a number of (what appear) to be unexpected results (i.e. poor relevancy, almost unrelated epinion/amazon/dealtime links, etc.)

I do, however, see light at the end of the tunnel. With all yahoo's new acquired crawlers, I think we are mere months away from a MAJOR shift in SE traffic. I, for one, will be glad to not have to be at the mercy of one player for the majority of my traffic.

[GoogleGuy]

Hey, I saw that people were looking for an update, so let me post what I've found out so far. First off, thanks to the people who sent a report with "indexpage" in it. Sometimes having more than one report makes it easier to investigate, so I appreciate the feedback. I was talking to another engineer earlier today and they did identify a potential issue that could have affected some of the reports. We have a binary-code push scheduled pretty soon (hopefully later tonight) for it. I'll keep you posted if I find out more.

[shawn]

Well,

I am not sure if I question GoogleGuys idenity etc.... I am sure he has a "full time" job that require his attention.

I stll have the "refresh problem" but it has changed slightly.

yesterday I would be number 9 then after a refresh of the page I was number 2 on this particualr keyword phrase.

Now today I am number one for this sarch phrase but after a refresh I am number 2. This does not happen everytime however.

Shawn

[kamikaze Optimizer]

As always GG,

Thank you,

Oh, by the way, will you be around during and after the code update, just in case we need you?

[drewls]

mipapage:

1) Yes

2) I moved servers, but that didn't affect the problem. It was happening both before and after. Plus I'm the host, so I control all dns records and can move things seamlessly.

3) Refresh and it's #4 for us. :) At least for the SERP we're most interested in.

5) We're in DMOZ, but still (after months) have yet to show up into the google directory.

5) Early April.

6) Yes.

GoogleGuy: Thanks for the info! :)

[Ltribe]

1. Yes, had both problems with the site

2. No host switching

3. Refresh and it's number seven

4. Yes, all are in the Directory

5. January in the Google/Directory?

6. Yes, the first to own the domain

[GoogleGuy]

kamikaze, it could go out pretty late tonight, or in the next day or so. It's not live now, so everybody should probably hold off on checking on this tonight. I'll be around off and on, but I'll plan to be around tomorrow more to give more details if I find out other stuff that people might be interested in.

[Napoleon]

I've got everything crossed that this fixes it GG.... but in the meantime for mipapage, the following applies to a number of sites:

1. Yes.

1. No. Stable virtual hosting

3. Not just #1, it happens for all sorts of positions

4. Yes, they are in there

5. All less than the 18 months

6. Yes, first owner

Given the tempramental nature of this, it will be an impressive bit of problem resolution if it's fixed. It's easy to check if you know for certain which sites are affected on which search terms, but I would imagine a nightmare if not. Here's hoping.

[Spica]

GoogleGuy, thanks for the update!

mipapage, I doubt that you will find a pattern through these questions, but here we go:

1. Did everyone here have both problems with their site?
2. Have you switched hosts lately? Are you virtual?
3. I've seen a lot of 'refresh and it's #1'. This is the case for me, how about the rest of you?
4. Are all of these sites in the Directory?
5. When was your site first into Google/Directory?
6. Were you the first person to own your domain (we weren't!)

1. Yes
2. Swiched host in May. No apparent problem with this move.
3. Refresh and I am # 3 (actually, #4 as of tonight :( )
4. Yes
5. Since end of last year
6. Yes

[GoogleGuy]

Just to narrow the focus down, I think #3 on mipapage's list is useful (for me at least) to know. If refreshing can put you where you expect to be, the upcoming push might help for that. Anyway, I'll be around tomorrow to talk to people more too.

[mipapage]

mipapage, I doubt that you will find a pattern through these questions...

I was more hoping these may spark some inspiration in some of the minds here - we tend to get stuck on SEO reasons when it could be some 'out there' reason. Anyway, hopefully the GoogleTweak sorts it...

Thanks for the updates GoogleGuy!

[kamikaze Optimizer]

<DELETE>

Spoke too soon...

[yonnermark]

1. Yes, its COMPLETELY disappeared. All 7 or 8 pages.

2. No.

3. No

4. No

5. 2 weeks ago. Has plenty of #1 and #2. Now all pages are gone

6. Yes

Note:
url: www.mydomain.com brings back the old cached page for the place-holder text that existed on my domain before my actual site was launched. That was about 3 weeks ago. Since then, my serps have been excellent but about 12 hours ago, my site's pages ALL dispappeared COMPLETELY.

Thanks for your presence Googleguy.
I'll check back for further updates :)

mark

[yonnermark]

panic over for me. My site is now back in the index/serps.

[mipapage]

Me too. My refresh problem seems to be gone! If it stays like this, Whohoo!

---

And a big thanks to GoogleGuy too... Glad we could help, and thanks for listening!

---

Hmmm, maybe premature. I had checked on ((three computers)x'many times') with clean caches, but the fourth failed... Of course right when I said "it's 'fixed', try it!"

<Second Edit>
It seems to be getting more and more consistent to the good for me. I wonder if the fix has to propogate across the servers, datacenters, thousands of little pc's and their thousands of little hard drives etc etc..
</Second Edit>

[edited by: mipapage at 1:48 pm (utc) on Aug. 14, 2003]

[Napoleon]

No change here yet.

[drewls]

None here either. Keeping my fingers crossed. :)

[WebFusion]

NO changes here...site is still MIA.

[c1bernaught]

Still seeing the same behavior......

[GoogleGuy]

Hey, it may take a litle while--I wouldn't worry yet..

[WebFusion]

Only movement I'm seeing now is that a couple of my competitors (who had gone MIA 2 weeks ago) are now gone from 4 out of 5 datacenters along with me.

The baffling thing is...one of them actually increased in PR, and yet his iste has started disappearing from SERPS as well...

Nothing like losing $400+ per day in income to get you motivated on diversifying away from the big G...

[zafile]

During the past months, many theories have been posted about the index page issue.

Yesterday, GG acknowledged that a Google engineer "did identify a potential issue that could have affected some of the [index page] reports."

I just read an article titled "Server breach raises Linux code worries" at [news.com.com...] Could it be any connections?

[edited by: zafile at 5:56 pm (utc) on Aug. 14, 2003]

[drewls]

LOL keep trying zafile. Keep in mind that after this VERY embarrassing week (some would argue that it has been a very embarrassing existence) for MS security, your trolling against Linux isn't going to do much good.

[GoogleGuy]

Nope, zafile, nothing like that. :)

[zafile]

Thanks GG.

By the way, I forgot to mention my problem with the index page issue was fixed. I'm pretty sure the issue was corrected when I lowered my keyword density a little under 5%.

My index page has been steady in the SERPs with some small variations.

I haven't experienced the temporal vanishing.

All the best.

[soapystar]

"keyword density a little under 5%. "

one day this will mean something but untill theres an agreed way to meaure density this doesnt mean a lot right now. 5% , is that with or without stop words?, are you counting a single word or a keyphrase? are you counting meta tags and title? page comments? image tags? etc. etc. there is no standard for density so 5% to you could be 10% to someone else or vice versa.

[zafile]

The "under 5%" figure covers only text on the homepage and per keyword.

HTML title, meta tags and alt tags aren't included in the figure.

The figure excludes the following words: a the but i or and of it he she her his on in an.