Google MAYDAY Update - SERP Changes May 2010

[pontifex]

< continued from [webmasterworld.com...] >

Now, as the dust settles from the last days and my panic is under control :->, I thought I share some ideas here.

Basic infos:

1. Nature of site: eCommerce

2. Type of link building: across the board
specialties: our sellers get subdomains, minimal link buying, majorly press, partners, etc.

3. Pages on domain: somewhere around 3 Mio. pages,
majorly product details and listing pages of products


History: Until Tuesday we ramped in around 200k Uniques a day - very strong long tail traffic, 65% of the landing pages are the product details.

What happened: First the site lost one PR point from 6 to 5 in the last update. Traffic went up, so I did not care at all, but maybe it completes the picture.

Tuesday, April 27th: solid and good traffic, but slightly lower than last week.

Wednesday, April 28th: first noticeable drop of around 10%

Thursday: 25% loss of traffic to Tuesday

Yesterday: 27% loss of traffic to Tuesday

Think this is the current status: Lost around 25% of power/traffic somehow.

--------------------------------------------

What happened?

Hard to say really, but here are some cents for the collective thought pool:

a) Keywords I monitor are still where they were, but these (around 20) keywords are not really LONG TAIL

b) Some merchants selling strong in the past weeks dropped out of the sky and vanished from our top sellers list: that could be the reason.

I looked here in this senseless long thread :-) and found the following quotes spark a clue:

I think the loss of long tail is more to do with Google's incredible improvements in Adwords relevance matching. - internetheaven

Well, my loss in traffic is majorly long tail - so that area fits at least.

Long tail traffic in most cases relies on internal link juice, this is how it works. - SEOPTI

There is so much insight in that one sentence, I love it!

And I can relate to it in my case:

The few top sellers vanishing from the list are providing thousands of niche products for very specific purposes. They lost the traffic and that whole picture falls in to place:

If this is the Caffeine update, IMHO the following happens while we type:

Deep content is gathered from domains not crawled so deep before. The amount of pages with long tail keywords doubles, tripples or just magnifolds and changes the game as a whole.

If internal link juice is now weaker or stronger: Link building has now a different meaning.

Link juice must be handled under a differen perspective.

Even if we can build up 1,000 named backlinks to a page, it looses its power on 3 Mio. children quite quickly.

If there are major authority sites unlocked by Caffeine with "before 100,000 pages" and after "10,000,000 deep pages" we have now

9,900,000 new enemies in the long tail....

That would explain a lot to me and I wonder what you think?

P!

< the active thread for this discussion is here: [webmasterworld.com...] >

[edited by: tedster at 7:44 pm (utc) on May 22, 2010]

[Dave_Hybrid]

I'm in the same boat, traffic dropped 50% in a few days, 100,000's of long tail k/w.

More content being found? More competition? Might be, might not be. To have all 100k of my k/w drop positions due to competition. Hmmm.

New weighting of internal links, possible. But then everyone would be affected in the long tail field, thus canceling it out overall.

Could be Google is now crawling deeper so found many new links, that might be throwing everything out of whack.

Could be Site Speed was just introduced on more than a "test" basis.

Could be lots of things. IMO best to wait for the dust to settle.

[1script]

@pontifex:
Welcome to the club!

Although you are not hit as hard as some of us (90% of my traffic's gone), you may be experiencing a similar issue with long tail traffic that was discussed in the March update thread and a few others:
March Update Thread [webmasterworld.com]
Collecting data about traffic drops [webmasterworld.com]

These drops appear to happen in batches, allegedly when some of the data pertaining to the affected site switches over to the the Caffeine-type storage/retrieval system. There was a batch of drops (or at least a batch of reports about drops) on March 15th, then April 16th, then April 24th and it appears that the time distance between the batches shortens as we go along - possibly a result of the completion of the switch.

I don't have a good grip on the issue myself yet but from compiling the data here and also looking at my own results for the last 1.5 months I think a couple of possible reasons stand out:

1. Internal link juice is distributed differently, though not necessarily just simply reduced. It could be that the order of internal links is important now, as has allegedly been the case with external ones.
   
2. Long tail matching has become if not laser sharp then pretty darn close to it. If you were easily ranking for a 4-word long tail having only 3 of those words in your copy, you now either need all 4 and not just in any order or you need support of a good (and allegedly external) backlink(s). Because of this perceived sharpness they have to pull up whatever crap they kept far away for lack of links for as long as the content matches well. I'm not saying all of new results are crap, I'm just saying that it's easier for bad pages to rank now as long as the on-page factors are favorable. Feels like 2003 all over again.
   

Anyways, like I said, I don't have a grip yet and I'm not sure if anyone here can claim that they do in this Caff/Decaff chaos, but any observation you can share, I'm sure will be welcomed here, at least by the fellow suffering webmasters :)

[dusky]

OK, still no joy since the 24th, still buried on page 4 and onwards, however, gbot is still at it like never before, I have to forward all alarms to my phone in case that server goes down because of it, the last time I seen gbot activity like this on our authority sites was Florida and BD updates. Some of those actually became large sites since and one was demoted and lost 90% of its traffic after BD and never recovered until this Mid April, when it started seeing some recovery, then bang now 90% of its traffic, mostly long tail disappeared.

So, to everyone with the "long tail traffic algo change" theory, I have joined your club. Further to that, is there a correlation between gbot massively increased activity and loss of traffic, that is, while the traffic is at 10-50% of its usual, DEPENDING ON YOUR SITUATION gbot is busy hitting your sites, I appreciate this one question survey!

[1script]

@Dave_Hybrid:

More content being found? More competition?

Same content, same competitors, have long been there in the index but never saw the light of day because our content was deemed better. Now, when ranked differently, they hit just the right spot.

New weighting of internal links, possible. But then everyone would be affected in the long tail field, thus canceling it out overall.

I don't think it necessarily cancels out. All sites are different, at the very least in the number of pages hence internal links, and so the effect may be quite different on different sites.

Could be Google is now crawling deeper so found many new links, that might be throwing everything out of whack.

I'm not sure if they crawl any different than they have been in the past but, with Caffeine infrastructure, they are supposedly able to take a better look at what's been there all along (and often is obsolete/erroneous by now). It may benefit older sites but it may not- if the old site had a lot of redirection errors/dropped URLs etc. - I have a couple examples like that.

Could be Site Speed was just introduced on more than a "test" basis.

In the aftermath of the announcement I've done some speed tests and looked at a variety of sites and came to a conclusion that this may be the last possible reason you need to be concerned with. Not saying it's not one of, just that you really need to exhaust all other possible problems (which is impossible in practice) until site speed comes into play. I'm talking about a site that does not take a whole minute to respond of course.

Could be lots of things. IMO best to wait for the dust to settle.

Don't lull yourself into passive waiting: just as success in Google breeds success via new links, viral promotion, bookmarks, feedback etc, the opposite is also true. Regardless of whether or not the site "deserved" the traffic drop in the first place, if you suffocate it without Google traffic long enough it will become irrelevant, obsolete and will further slip in ranks via "natural" ways - link attrition etc.

[1script]

dusky, my answer to your one-question survey is no. I'm not seeing any easily discernible correlation on any of my affected sites. A few people responded to my requests for more data via PM and none have indicated any abnormal Gbot activity. Activity varies, mind you, but nothing on the scale you're describing that borderlines with DoS attack. In fact, Gbot has always (well, last 5-7 years) been able to scale back when they saw performance troubles. Unlike some crawlers of other respected SEs who's name starts with Y. I mean, one bad night of really heavy hits - maybe - but consistently for a week - never seen that.

Speaking of DoS attack, have you double-checked the IPs? On some days I'm seeing 60% of what calls itself a Googlebot coming from IPs that do not reverse-DNS to anything remotely close to Google, Inc.

[steerpikegg]

Dusky,

That's exactly what has happened to my site. 50% loss of traffic and constant hammering by googlebot.

[member22]

How can you see that google bot is crawling your site like crazy ?

[dusky]

1script, yes all IPs are of the crawl-66-*-*-*.googlebot.com type and if others were used they are all RDNSed to G* mountain view.

So, to everyone with the "long tail traffic algo change" theory, I have joined your club. Further to that, is there a correlation between gbot massively increased activity and loss of traffic, that is, while the traffic is at 10-50% of its usual, DEPENDING ON YOUR SITUATION gbot is busy hitting your sites, I appreciate this one question survey!

The list starts with:
dusky,
steerpikegg

steerpikegg, myself are definatly on agreement on the above, any others! I'll wait for few replies so I don't have to quote myself over and over again and update the list here.

Although my hand is on my heart and sometimes my two hands are catching my sunken face, I am trying to look at this positively, meaning some good will come out of this, likely to be they are deep crawling the millions of pages and that takes time, whereas before they stopped at 100k for example and dumped the rest because of shortage of infrastructure space, a problem they hopefully solved with caffeine. In the meantime and while re-indexing and recalculating PR, those sites are put on hold somewhat (as I pointed out before) for their long tail (stored in the new structure waiting to be released once all is re-indexed), hence the gbot high activity!
For me, this happened the third time in two months (March-April), twice they recovered after less than 5 days, now its the seventh day and not yet!

As long as you know there was nothing fishy done on those sites and completely white hat, you can add yourself to the list of the shrinking long tails theory, if that's the case for you!

I did not call May Day on this, but looks like it literally came whether I liked it or not, see the date of this post :(!

[dusky]

member22,

How can you see that google bot is crawling your site like crazy ?

Just look at your server's access stats logs every few minutes, refresh every minute or so, for example using cp*nal's Latest 300 Visitors Stats or similar which gives IPs, referrers and user agents etc. You can do it on a command line terminal, but I am lazy, can't remember, is it grep whatever...in the webserver's stats or access logs, something like that.

[Dave_Hybrid]

Bot activity is normal for me 20-30k pages daily

Large 600k page forum, 5 years old, 20k uniques a day, whitehat.

[pontifex]

dusky:

I did not call May Day on this, but looks like it literally came whether I liked it or not, see the date of this post

well, could we officially name this update MayDay? Caffeine is not the proper name and Matt is not unveiling anything... so lets take back at least the name control from them and call this update officially "MayDay"!

Tedster, engine, brett? Some weight on this, please - i think it is more than proper naming :->

P!

[giviz]

Hi guys,

Same long tail problem for one of ours websites (old website with 5kV a day)
We have several websites, all french and all with the same architecture.

One of them (the biggest in traffic) have been hit by the long tail problem between the 14th and the 15th april.

What I can say is :
- Only internals pages have been hit by the LT problem, what I mean is that my home page, whose got a pretty good LT has not been hit.
- 90% of the internals pages have been hit, they loose all their positions and traffic and now have only very long tail keywords
- On that internals pages, traffic drop by 80%
- I can see that some internals pages have been hit on the 22 march and not on the 14th april, but that's a minority
- Some internals pages have not been hit, all have one particularity in common that they share with all others pages on all ours others websites that are based on the same model, same activity but different niches.

All pages that have been hit are long text content model (2000 words and more).
All pages that have not been hit are short text content model (near 600 words).

Google webmeisters tool tell me that crawling of my hit website has not changed.

So, what does I think of that :
Hit pages are made with a mix of second and third level pages in our old architecture.

We had :
widgets (page 1, level 2)
blues widgets (page 2, level 3, link form level 2)
reds widgets (page 3, level 3, link form level 2)
...

Now we have changed that a long time ago and have :
widgets (page 1, level 2)
widgets blues (page 1, level 2, h2 paragraph and his content)
widgets reds (page 1, level 2, h2 paragraph and his content)
…

For information, level 1 is the home page.

I think that that architecture that works really great before Caffeine no longer works now. As ours others pages based on the same model, on several websites are not affected and are all without the h2 and paragraphs content, I think smaller pages with a real unique topic are now the new choice of Google.

Hope that can help someone and you can look if your situation seems to be like mine.

I will edit all my affected pages to remove all content that is not very closed to the primary topic of each page and see if things goes better.

What do you think about that ?

Sorry for my bad english, I do my best.

[dusky]

giviz, eh bain alors, soyez le bien venu ici :)


I can safely add you to the list of long tail sufferers when I get back to it with a couple more people, as I can see your site/s more or less seem to have their data you provided almost fitting the equation.

I swear I did not swear here (for those who always missed Miss Jackson's French class)!

Au revoir alors et bonne chance!

[dusky]

The May Day update, I'd love that especially if our sites recover with a big bang soon as a result of this update, nonetheless, May Day it shall be named in my book, and whara-day that was we shall all remember it by!

[tedster]

Another short survey question for long tail problems - how many words long, on average, are the phrases that lost their traffic? Are they more than 5 words long?

I'm wondering if Google has made a change in their phrase-based indexing approach - something that the new Caffeine infrastructure makes feasible. Recently there has been more patent activity in that area.

Indexing of phrases is typically avoided because of the perceived computational and memory requirements to identify all possible phrases of say three, four, or five or more words.

For example, on the assumption that any five words could constitute a phrase, and that a large corpus would have at least 200,000 unique terms, there would be approximately 3.2.times.10.sup.26 possible phrases, clearly more than any existing system could store or otherwise programmatically manipulate.

Index server architecture using tiered and sharded phrase posting lists [patft.uspto.gov]

In other words, until recently queries for long phrases may have had something like "best guess" results using some secondary signals -- but now Google has the infrastructure to index longer phrases much more directly.

It's a brainstorm idea at present, and not a solid "statement of fact". But hey, we have to start somewhere.

As a side note, thanks to Google I now remember how to spell caffeine!

[edited by: tedster at 7:05 pm (utc) on May 1, 2010]

[dusky]

pontifex

WOW, looks like tedster (or who was it) has bought your MayDay update idea (in memory of my fallen sites?) by naming the thread MAYDAY Updates or is it normal!, if that's the case, it's good to give up caffeine and sleep better!

[tedster]

Yes - it was me that renamed this thread. I wanted to honor the observations currently being made - especially because the reports of search traffic drops are coming from highly respected members who we know aren't just fooling around.

Something very real has shifted at Google, but apparently it takes a certain type of webmaster/website to notice it - significant long tail traffic closely monitored in detail.

A lot of the sites I work are heavily dependent on branded query terms, and they are not showing this pattern. The presence of their brand name, even in a long tail search, carries the day for them.

[edited by: tedster at 9:01 pm (utc) on May 1, 2010]

[giviz]

giviz, eh bain alors, soyez le bien venu ici :)

Merci ! Actually I'm in the USA (Miami), so I can speak english ;)

I can safely add you to the list of long tail sufferers when I get back to it with a couple more people, as I can see your site/s more or less seem to have their data you provided almost fitting the equation.

Unfortunately, I agree.

Another short survey question for long tail problems - how many words long, on average, are the phrases that lost their traffic? Are they more than 5 words long?

My pages used to rank on keywords about 2 words, 3 words and a lot of combinaisons with more words. But there is a lot of 2 words (I'm french, less traffic so LT start earler in number of words I think).

Now My pages rank for 3 to 4 words at less and 2 words keywords with mispelling.

[dusky]

Please give tedster a round of applause (mind the coffee cups on your desks) at least to liven up the place amidst the chaos and this update storm.

In other words, until recently queries for long phrases may have had something like "best guess" results using some secondary signals -- but now Google has the infrastructure to index longer phrases much more directly.

I think you cemented what all of us here including yourself were getting closer and closer to believing, too many similar indicators and consequences hitting many respected sites with identifiable symptoms can't be all wrong. Makes a lot of sense, more space means more choice of data combinations, hence the MAYDAY infrastructure update, yes I know MAYDAY meant for the update (caffeine infrastructure update) so was BD or Florida's infrastructures.

[1script]

In other words, until recently queries for long phrases may have had something like "best guess" results using some secondary signals -- but now Google has the infrastructure to index longer phrases much more directly.

What's secondary to content? Links! Links are out, content's in. Altavista won. Matt Cutts lost and retreated into self-imposed exile in Japan or Thailand [mattcutts.com] - whereabouts unknown. Google is to be renamed to Altavista promptly. Look for public announcement on Monday.


I know, I'm one month too late with this, would've been more appropriate on the Fools Day :)

[drall]

Digging through our stats it appears it is really effecting our 2 - 3 word phrases. Traffic now down an additional 10% since Wednesday.

The steps down have happened at certain intervals that match almost perfectly to 1scripts dates. As a whole after taking seasonality into account over 10 years of data I can now state we are down 50% in google traffic for 2 out of 3 of our sites.

I dont know if this will help in figuring this out but the site that is most effected stats are
PR7
13 years old
Over 400,000 IBLs including links from Google, Yahoo, US Govt, State Govt, Technology giants.
Site runs direct advertising and does not use adsense or 3rd party ads.
No GA running, webtrends 9 instead.
No substantial changes to design or seo.
Security audits show site and server are clean.
DNS clear

Nothing has changed on our end to the best of our knowledge.

Dont know if this will help either but our smaller niche pr 5-6 sites that we barely work on and have almost no links are seeing absolutely no changes in traffic through all of this. Granted these are small sites with only 1000-4000 visitors a day but I would think they would see something to but they are not, not even a 5% deviation.

Just seems like our powerhouses are getting whacked. I just dont get it. Very upsetting, trying to remain calm cool and collective and look for details while eating xanex like its candy :)

[dusky]

drall is now member of the long tail club, sit down drall, can someone get a glass of water, no coffee.

Yes, unbelievable, giant, well established sites getting whacked, that explains the MC hammering, then going away to be on Tea instead.

Sorry MC, I liked the way you found the time to answer my mail before, a good listener (outside this site), but I am not happy and one has to speak out!

Surely this has to be temporary, in fact I am 99% sure as I said earlier it is temporary, if it's not, I'll invest what's left on B*ng's shares, well MincroS's as the decade 2000-2010 will be known as G* success years, not after, and MAYDAY would be the precise day!

[1script]

PR7
13 years old
Over 400,000 IBLs including links from Google, Yahoo, US Govt, State Govt, Technology giants.

drall, you said before that your number of backlinks will make me cry. It did.

But just so you don't feel all that lonely there at the top, if it's of any consolation, all my sites fit your profile of smaller niche sites. Most are hit as well.

I'm going to have to re-process my logs for before the crash and after to fill in here per tedster's request, hope to have some data in a couple days. Don't want to just speculate - the implications appear to be too important.

[Bigwebmaster]

Just wanted to quickly state that I am in the same boat, most of the traffic on the sites I run are also long tail, very established sites running since about 2002 and we have had about a 10% drop in traffic since the beginning of April, and most of the drop started since this last Thursday.

[tedster]

Don't want to just speculate - the implications appear to be too important.

Agreed - this could be a big change in the SEO terrain, or it could be a temporary side-effect of some new approach that needs to be tuned up better at Google.

I also hope for the second option, because it seems unlikely that Google would intentionally whack major websites with no guideline "issues".

[dusky]

IMO, this is what's probably happening (I said this before), half of the data is on the new IS, the other half is still on the old or vice-versa, the algo is pointing and trying to retrieve / fetch search queries results from the new IS, however, half of the data is still somewhere else, so it fetches results based on what the databases have, and they have a lot less. The data that is still to be moved and awarded PR and IBLs/EBLs is mostly deep end-of-branch pages which are the bulk of the Internet, while homepages and second in PR command pages are likely to be the short tail well established pages, and they are mostly fine because they were moved first.

I wrote a crawler once while back and when doing the crawling, I fed it first the authority sites homepages mostly from Y and Dmoz directories in addition to Alexa's top 100k, and let it work and drill downwards through them before fetching links posted on them. Feeding that into my infrastructure was by page hierarchy importance, the deeper the page, the longer it takes for the crawler-indexer to work out where to throw / place it, hence if one were to search that particular site, they will get results that are not necessarily worthy of user's satisfaction, because better results aren't yet inserted into the database and would take days, so my small-man algo at the time did not have any choice, you offer answers based on what you find as records. OK, those were the days when you can have 100m index and think you are a big shot, if the scenario is true for sure on G*s case, no surprise it's taking months and getting more and more noticeable, happening to more sites, the whacking is even harder now because we are in the middle of the MAYDAY update which can still go on for longer. MAYDAY is certainly the milestone as more people than ever felt the heat, just like BD and Florida, though those were not one day dump-all either!

This is why I think is deliberate and they can't wiggle their way out of it, it's the sacrifice to pay (we pay, to probably make them pay) for moving to larger IS.

[maximillianos]

We are seeing the same this week. We are down about 15% in traffic if not a little more. 10 year old with very heavy long tail organic traffic that has been steadily climbing for years as our content grows organically.

So a drop like this is not something we see often. In fact we have not seen one like this in 2-3 years.

[arizonadude]

I also hope for the second option, because it seems unlikely that Google would intentionally whack major websites with no guideline "issues".

Why does that seem unlikely. For as long as I've watched Google going back 10 years now, every single change produces results that whack websites for no reason. For what ever reason, those sites are not looked at favorably by the new factors that are currently at play.

Sometimes, more get wacked than other changes and this looks like one of those times.

[tedster]

You may well be right. My first idea was based on misunderstanding the length of the queries that people were calling "long tail".

2- and 3-word phrase are what I think of as the "fat belly" - it's in between the "head" (trophy) phrases and the "long tail", which is more like a 4- or 5-word phrase and beyond. (That's a 4-gram or a 5-gram, in IR talk.)