Big sites suffering no title / no snippet in SERPS

Is google penalising big sites?

[cyberax]

As of around a week ago it seems that a lot of big sites, particularly some UK search sites, have lost a lot of their indexed pages in Google.

The pages haven't been totally removed, instead it seems that the pages exist in the Google SERPS but they have no title and no snippet and therefore no longer appear for any searches.

At first I thought this was some sort of penalty / filter to remove some of the controversial search sites from its index, but it seems this applies to other large sites, e.g. dmoz. I would estimate that dmoz has had around 200,000 pages "nuked".

Has anyone noticed this phenomenon on any other sites?

[ThomasB]

I'd still say we should focus on calling it a filter/penalty and not a Google problem. They have enough money to do whatever they want. If not, they shouldn't have problems getting money from the bank, I guess.

[Yidaki]

>I'd still say we should focus on calling it a filter/penalty and not a Google problem.

I disagree. I observe the behaviour now even with a very new site. Launched around March 20, all 90 deep pages indexed 2 weeks later (no top rankings though). This morning 90% of the pages don't have a snippet / title - url-only. Referers dropped for the few phrases that had good rankings. This site is clean - nothing to filter, nothing to penalize.

I wouldn't call it a Google Problem but just the way google behaves currently. Recrawling allready known pages - Googlebot seems to remove them from the index first and add them back in later. I'm sure, my pages will show up again shortly.

I'd say no filter, no penalty.

[Abigail]

I am almost afraid to say anything that might jinx me, but, it looks like my pages are starting to come back, titles and desc, at about 30 per day for the last few days. I have also made an interesting observation, not technical, but images are appearing on a regular basis in the new image search part of the index, and interestingly enough, they are images off my disappeared pages - so is this all some sort of reindexing exercise that is causing all this grief?

[ThomasB]

Yidaki, couldn't it be that it's just the normal in and out of the fresh index? If you see it on sites online longer than 6-8 weeks I'd agree. But you still don't know if "G is broken" or if it's a penalty. It's also possible that the site you mentioned has the penalty.

[Abigail]

ThomasB:

FYI - My entire site over 1500 pages has been online for over 4 years and enabled an average PR5 on the pages and 6 on a lot, I have a few thousand backlinks from other sites.

[asgdrive]

My company has over 500,000 auto parts on our site and over 90,000 pages. Our site had great page rank and all pages did very well until the first of January when Google started to delete pages. no matter what we do we can not get above 5,000 pages.
We also tried new sub domains, dividing the site into 20 make categories with much less data reducing the Pages to less than 500 per domain. The new pages were indexed within one week (4 weeks ago) by Google and stortly thereafter were completly dropped with only the URL listed (no title or description). This has reduced our revs by 90%.. All of our competitors are suffering the same issue. This seems to be only effecting large sites. Your guess is a good as mine?

[edited by: Marcia at 5:23 pm (utc) on April 27, 2004]

[ThomasB]

asgdrive, do you know have 2 pages with the same content on the same domain? Maybe duplicate content is the reason. Would check that and do a 301 from the old structure to the new one.

[djgreg]

Now one of my sites has also be affected by this problem.
The site was #1 for a very competitive search for years, had PR7 and a huge amount of backlinks.
1 week ago the index page started to fall down from #1 to #23. I wondered how this could be because the sites above me do not have as much backlinks and PR as my site has, but I did not think of it as an penalty or problem.
Yesterday my sites vanished completely from the results. All pages got PR0 and the index page got PR5. But you can't find it on your key term in google.
What really confuses me is the PR5. I have about 5 Links from PR7 sites and one from a PR8 site, so why is the index page PR5 now?
Additionally Google shows no backlinks at all for this site. I had about 500 before this "problem."

I don't believe this to be related to the index capacity. Why should my index page get PR5, and why should my index-site be removed from the serps when the problem is capacity. These are 2 things which have nothing to do with each other. If they have a capacity problem, I would think that they remove a certain percentage from the sub-pages of every URL. But why should they stop counting backlinks or decrease PR?

greg

[TinkyWinky]

djgreg,

Just looked at your site - are they all links at the foot of the page to 'your' sites or people you have linked with? Are there reciprocal links on all of those sites back to you? I know someone else who doing exactly that got a very similar penalty and he had a PR7 too.

Just a thought.

[djgreg]

Hi,

none of these sites are mines and most of them have set up links to my sites but not all.
They are all Quality Sites related to the topic of my site. Some of them are regional offers, some of them have offers I can't give my users on my site.
Furthermore there are only 40 links to other pages. I know plenty of sites which have more links on their index pages.

greg

[g1smd]

I really haven't got time to read a bazillion post thread, but I have just noticed that every ODP category that comes up in backlink searches is formatted as URL only; with no title or description.

[djgreg]

g1smd: yes that's right, we have seen that before.

greg

[sem4u]

I've lost most of my titles and descriptions for the dynamic pages on one site.

Anyone got them back or lost them again?

[BeeDeeDubbleU]

I don't think that this is exclusively a big site problem. It may be that it is just more noticeable on big sites because of the number of pages that are being dropped and also because they are more closely monitored.

I uploaded a new site about five or six weeks ago. This has only 15 pages. It appeared after a few days with great results then almost immediately disappeared. Yesterday when I checked, the only page that had both title and cache was the home page. Today all the pages are once more back in but it is still not ranking.

I also had the problem with an 80 page site. This has also now recovered its PR, page titles and descriptions but it is also not ranking?!?

[Yidaki]

>Anyone got them back or lost them again?

Yes, mine are coming back. Today, only ~50% of the titles/snippets are missing (for the small site i mentioned above). Also the url-only listings for my big sites decrease slowly.

So, there's hope ... obviously not a penalty.

[djgreg]

Obviously not a penalty

But what is it?

[kosar]

has anyone come to a conclusion on how this problem gets fixed? i submit my site at least once a month and googlebot is always on my site, this has never happened before.

[jimshu79]

Well you definitely dont have to submit to google once a month, actually, prolly never.

But back to the topic, yes if anyone has an idea, my 3 site are losing titles and desc, no serps...help...

[AjiNIMC]

I hope this gets fixed soon. Will re-submitting to google will help.

[jimshu79]

no because as mentioned all over this site, google is just as happy by picking up your site following links. getting more links to your site would be a good step. but if you were indexed once, and not banned, you'll get spidered again. but this sucks...

[Abigail]

Well - I was coming back slowly and was getting optimisitc and this morning the rug was pulled out from under me again - I was up to 435 (out of 1560) back and this a.m. down to 400 - would sure like to know what's going on - it's been a very stressful 4 or 5 weeks now!

[kaled]

I'm feeling mischevous - it's not that long ago that Google passed the 4 billion indexed pages mark. Is it possible that someone compiled a program with 32 bit integers instead of 64 bit?

It would certainly make me laugh if this turned out to be the cause of the problem.

Kaled.

PS I added about a dozen pages to my site about a month ago. They were duly indexed and dropped to url-only listings about ten days ago. My site is a tiddler so I can honestly say that size doesn't matter.

[renee]

kaled,

it's not a program compile problem though related. it is a an index problem also limited to 32 bits. to overcome this, google seems to have several buckets (db):

1 - main or primary index that contains the fully indexed pages (limited by 2**32). this means that for every new page that gets in, another has to be dropped. google has to add new pages otherwise it becomes obvious that google has this capacity problem.
2 - supplemental index (to relieve the main index capacity problem) that google refers to only if too few results are found from the main index
3 - the "url-only" bucket that google "never" got around to indexing (because they're out of index space!). note that the "url-only" pages seem to be dynamic from update to update. i would say google simply randomly decides which pages to leave out. this bucket is used by google only for site: or url: queries.

note that all the google problems that have been reported recently can be explained by this model. It's a totally random process which has us all scratching our heads!

does anybody have any evidence that the total number of pages in the main index (i.e. not including supplementals and url only) is over 2**32?

[ThomasB]

[google.com...]

Results 1 - 100 of about 5,800,000,000 for the

2^32 = 4,294,967,296

Is that enough?

[renee]

"http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=the"

all i see is this:

Results 1 - 10 of about 31,000,000 English pages for the [definition]. (0.27 seconds)

looks like google is playing tricks with you.

[renee]

thomasb,

since i cannot reproduce your query, could you please check if it includes supplementals or "url" only pages.

i've done regular queries before where google unpredictably also includes supplementals or "url" only pages. i say the total numbers shown by google is unreliable.

i still maintain that all the weird google behaviour we're observing are attributable to this.

[Monkscuba]

Try searching for "the" with Safesearch off.
5 billion plus results.
Safesearch on, only 28.3 million

[ThomasB]

I get 994 results (going to page 10 with num=100) and I can't see any supplemental results there.

Results are always limited to 1000 results, so I doubt you can check it.

[renee]

monkscuba,

i have safesearch off and i still get the following:

"Results 1 - 10 of about 28,400,000 English pages for the [definition]. (0.14 seconds)"

is there a way of specifying safesearch off through the query instead of through the preferences?

i also tried advanced search with filtering off and i get the following:

"Results 1 - 10 of about 29,100,000 English pages for the [definition]. (0.16 seconds)"

note the different totals.

i know it to a be fact that the serps can, but not necessarily, include supplementals and/or "url only" entries. this means that for serps > 1000, you cannot conclude that the serps do not include supplementals and/or url-only entries. so we still cannot disprove that google's main index is limited to 2^32!

[g1smd]

One of your searches has a stray quote mark" at the end of the query.

Look again at the posts above.