Google Updates and SERP Changes - March 2010

[drall]

< continued from [webmasterworld.com...] >

I have been watching all of my competitors which pretty much live off of large amounts of longtail traffic loose overall traffic the past 4-6 weeks and that includes us.

I dont know whats going on but for big sites in my section of the web site page count has been dropping at a constant decay and new content is coming in very very slowly and not nearly enough to counter the drops.

Overall traffic seems down about 8-10% for all of us and those drops are graphing perfectly to the drops in page counts since mid January.

At the same time the crawl graph in WMT and my local stats package show a flatline since mid Jan with no deep crawls and about a tenth of usual crawl activity.

All of this for us and our competitors started mid January
1. dropoff of the googlebot deepcrawl with a flatline stable crawl at a much lower rate.
2. daily reductions in pagecount across a dozen sites by exact same percentages
3. similar gradual traffic declines matching page count falls
4. uber slow new page caching and ranking

When I see this happening to not only us but many pr6-8 sites in my sector it gives me comfort to see that it isnt only us but also concern because it isnt only us.

Just going to keep chugging along, not much you can do but sit back and watch this stuff unfold.

[edited by: tedster at 5:49 pm (utc) on Mar 1, 2010]
[edit reason] split from earlier thread [/edit]

[foolsgold]

Whitey, nothing too much that we can see in terms of brand activity affecting Vince.

[Whitey]

Seems way to random for me - i'm scratching my head .

[tangor]

Looks like I might have to try Google for a day or two to see what's going on. I made the switch to Bing several months back and have not regretted it.

[Freedom]

It's showing in results already with a cache date of yesterday. So there certainly isn't a delay with new pages."

No offense, but this has been a problem with WebmasterWorld members ever since it's beginning. They assume that because something is happening to them, it must be happening to everyone else.

Problem is, there is a fresh bot for blog and news articles that updates daily, and then there is a deep crawl bot. It's the deep crawl bot with cache dates going back to Jan 1-9, and no new pages being indexed that I believe is the problem for a few of us here. Personally, I think it's wide spread because I have been checking out the caches on dozens and dozens of other sites the past 3 weeks. I have a PR6 website with 100s of interior pages that date back that far. It's very strange, and not normal, IMO.

[vordmeister]

There's a thread about the indexing which was started on 1st December and still ongoing: [webmasterworld.com...]

I didn't realise there were 2 discreet bots. That might explain how a forum can be indexed very quickly, but other new content on the same site can take months to be indexed and even longer to show up for anything (if at all).

[gouri]

Everything is just so slow on G right now, and I haven't had a deep crawl since January 9. The pages are getting cached very intermittently, a couple of pages are crawled here and there, but not a deep spidering.

The pages that are being cached very intermittently, have they not been changed for a while? That may be why the cached page is not being updated very frequently. Google may be using their resources for pages that are being changed more often.

[crobb305]

gouri,

They are established article pages, but my site is under 110 pages and I am always making tweaks. For instance, about a month ago, I discovered and removed some dead links on those pages. Google has yet to deep spider and recache. Beyond things like that, how often should an article page change? I can understand a news site or blog getting hit frequently because of new content that hits the homepage, but on the permanent url for an article (on a news site or a blog), it doesn't seem that the content would change any more/less than mine. Those articles/posts become an established part of the site.

By the way, I am just making an observation, I don't mean to give the impression of whining. In general, I think it's interesting that many of us have seen something similar since mid January, and I assume they are focused on Caffeine.

C

[barretire]

I have noticed that for the past week crawling has slowed to almost a stand still.

[crobb305]

Another example of the staleness: On January 12th I discovered a domain setting their CNAME to mine, and getting indexed with my content. I set up 301 redirects that same week, to force requests for their domain to my canonical. Google STILL has their domain indexed with my content. Another site that was using my logo was taken down in mid January (serving 404 for 6+ weeks), but is still indexed. Granted, I could submit the latter to the url removal tool, but I shouldn't have to.

[tedster]

I have noticed that for the past week crawling has slowed to almost a stand still.

Even for the biggest companies whose data I can see, googlebot shows up less than slurp and MSNbot - certainly not the usual state of affairs.

[freejung]

I just checked, and my total number of indexed pages is down from its usual value by over 50%. I'm not sure how long this has been going on, and it doesn't seem to be causing a significant drop in traffic. Presumably the pages that were dropped didn't get a lot of search traffic anyway.

[gouri]

They are established article pages, but my site is under 110 pages and I am always making tweaks. For instance, about a month ago, I discovered and removed some dead links on those pages. Google has yet to deep spider and recache. Beyond things like that, how often should an article page change?

That's a good point about article pages not really changing.

It appears then that Google has just chosen to slow down the cache updates for websites. If you are always making little tweaks, I think the cache updates should be more often and not what you are seeing now.

As you were saying, maybe Google is really focused on rolling out Caffeine and cache updates seem to have taken a little bit of a backseat.

[Freedom]

Yep, Google Cache is so 2009 and MSN and Slurp are so 2010.

Literally.

BTW, I've made big tweaks on pages, and google still is not updating the cache/index on mine, or many other websites that seem clean.

[mromero]

We updated the description meta tag on one of our websites in January as it was old. G picked it up within a couple of days and displayed it on the search results. Since last week the old description is showing again - copied direct from the website's DMOZ entry which is like 2 years old.

[BillyS]

No offense, but this has been a problem with WebmasterWorld members ever since it's beginning. They assume that because something is happening to them, it must be happening to everyone else.

I absolutely agree with the above statement - except for the typos. ;)

No one likes it when there is a counter-argument to their findings. I’ve been around long enough to have read all of the “Google’s broken” threads at least twice. I’ll do everyone a favor and retire from Google Search News. It will be far less frustrating for everyone involved.

[Seattle_SEM]

Even for the biggest companies whose data I can see, googlebot shows up less than slurp and MSNbot - certainly not the usual state of affairs.

That is really reassuring Tedster! Notice what this guy posts, he's always spot-on.

[vordmeister]

It's not just tweeks that are slow to be indexed at the moment. Try launching some new pages.

Some especially good pages in a section I launched back in November are still not in the Google index. In the past I would have expected them to be searchable for unique terms in Google faster than Bing or Yahoo.

It's surely due to Caffeine not being launched when intended, but I don't understand why Google are tolerating the lag when Caffeine might still be months off.

[SEOPTI]

I'm thinking about moving my pagination above the content to get a better crawl. Until now pagination below the content worked fine but with this mega slow indexing I will have to test it ..

[crobb305]

One of my other issues with the extremely slow crawling is that a site I work with was hit with -40 to -60 in mid Jan (seems to vary day to day, datacenter to datacenter). I have mentioned this in the previous thread so I won't go into details. But, if the filter was imposed algorithmically, then it COULD get lifted algorithmically provided that a deep crawl discovers the on-site changes/cleanup. However, there has been NO deep crawl since January 9th so the changes I have made basically have gone unnoticed. I have no experience with this filter, but I have been hoping for an algorithmic removal. I imagine the filter itself implies a loss of trust which may have affected the crawl rate on this particular site, but I have observed very slow crawling on other sites. Obviously it may take a manual intervention, but I keep hoping and waiting :)

C

[Freedom]

Typos? Where's my typo? it's to its?

[AG4Life]

Anybody else getting only the non supp number of results returned for certain sites for the standard site: search?

For example, site:example.com returns the exact same number of results as site:example.com/*

Also, as part of monitoring the steady drop of some pages that I want removed from the index (see this thread [webmasterworld.com]), it has pretty much stopped dropping since around Feb 17/18. It was 221 results then, and it's now 217. It has never been this slow before (for example, for the week prior, it had dropped from 300 something to 221, was about 800 in Janaury). Googlebot activity has slowed as well, as noted by others here.

[SEOPTI]

Maybe with really small sites up to 1000 URLs it's the same for "site:www.example.*" but when you do a query on WebmasterWorld URLs and different larger sites you will see it is a completely different number.

[TheMadScientist]

Typos? Where's my typo? it's to its?

It's (possessive) is correct...
It's (it is) not supposed to be its (it multiple times).

I can't find it either.
I've read the stinking quote at least 5 times...
Anyway, enough English review for the day.

Traffic was up last week on one, dropped back to regular or slightly up over the weekend and yesterday, is up again today.

[crobb305]

a little off topic, but I am starting to see the new Google search layout/design again. I don't know the name of it, but I really like it. Very clean.

[AG4Life]

Maybe with really small sites up to 1000 URLs it's the same for "site:www.example.*" but when you do a query on WebmasterWorld URLs and different larger sites you will see it is a completely different number.

I tried and got the same number of results back for a site that used to have 70,000+ main+supp pages, and 25,000 non-supp. Same with my website of 15,000+ pages.

But as I type, it's gone, and it's back to being different results for "site:example.com" and "site:example.com/*"

[Freedom]

I was bending my brain trying to find that typo. I was thinking, what does he see that I kan't? ;-0

I have a degree in English and another in Journalism, so I was really looking hard to see it.

[helpnow]

Anyone else noticing 3 distinct datasets via different proxies right now? 1 dataset is brand new (variants of it were seen 6 weeks ago, but it has not been active for the past month at least), 1 dataset has been active for the past 5 days, and the last dataset was from last week.

[imbckagn]

Caffeine status [webpronews.com...]

[aristotle]

The internet is growing at a rapid rate. Millions of new pages are created everyday just on social networking sites alone. This is one reason Google decided to implement Caffeine, to increase their data handling capacity to keep up with the growth. But Caffeine hasn't been implemented yet, and in the meantime Google may have to drop some pages from the index in order to make room for new pages.

[cangoou]

... the question would be: How to make sure that G is not dropping my page (or pages linking to me)? This is exactly what I see: Old sites went down -20/-30 and are now coming back to Top 10 without doing anything. Just as if G forgott some of the incoming links for a while.