Pages getting indexed. is there a slow down?

[Onders]

Google is a very regular visitor to our site, we're in google news, and we have a lot of quality backlinks - and we rank well.

Recently however some new pages that we've put up have bizarrely not been indexed and a search for the specificc URL on google shows nada.

I have heard that google doesn't index absolutely everything, but is this becoming a more common trend? Is there potentially content (like keyword overloading?) that means that Google won't index it?

One new page which has got an awesome amount of unique content has been linked to from our homepage, and was made live about 6 weeks... and still hasn't been indexed...

Any thoughts? Thanks

[tedster]

Yes, I've heard several reports that indexing of newly spidered pages has slowed down in recent weeks. Whatever is going on with Dewey [webmasterworld.com] seems to have taken a toll in other areas for a while.

[santapaws]

pages getting lost too, new pages indexed quick enough, show for a couple of days and then poof! lost in action!

[SEOPTI]

Yes, there is a slowdown.

[fishfinger]

They're too busy indexing urls via forms and unlinked code (which no-one actually wants) to index them via something as mundane as a good solid site-wide <a href=""> tag ;)

[TheMadScientist]

I've notice the slowdown too.

I even resorted to submitting a couple of pages to see if it had any effect, but no luck.

[SEOPTI]

For the first time Yahoo is crawling more URLs than Google, congratulations. It seems Yahoo improved their crawling capacities.

[Onders]

I've had a similar thing with pages being indexed, getting traffic and then the page disappearing from the index. There's nothing wrong with them - similar style and format with completely unique, and not overly optimised very good text.

Submitting a sitemap regularly to google - could this help? I've heard that submitting a siteamp "does not increase the speed of indexing" - but perhaps it could help?

[cgrant]

Up until a month ago my site was indexing new content at a healthy 20-30 pages per week. Since this "dewey" nonsense, our indexing has not budged in 3 weeks.

[minnapple]

Lately, I have noticed that the "cached" results have become fresher than the search returns.

[dataguy]

Glad to hear that I'm not the only one experiencing this... I have a report which shows new pages from the past 7 days which are receiving Google referrals. Used to run about 400 pages, now it's about 140.

I saw that Tedster posted in another thread that a lot of the regular processing functions seem to be taking longer than normal now. Care to elaborate on this, Tedster?

[fishfinger]

Submitting a sitemap regularly to google - could this help?

I wouldn't think so unless things have changed recently. Sitemaps only aid discovery (if your site is huge and urls hard to reach).

They don't influence indexing - as far as I can see that is governed by PR and uniqueness.

[filemakerguy]

I've experienced a slow down in the index rate as well. I can see Googlebot hit the pages but they are not appearing in the index as fast.

[System]

The following message was cut out to new thread by tedster. New thread at: google/3629317.htm [webmasterworld.com]
1:45 pm on April 17, 2008 (EDT -4)