Comparing what's getting indexed to what's not...

Googlebot visiting lots of old friends but not making many new ones

[stcrim]

From several threads here and personal experience, G-bot is continually visiting and updating sites and pages that were in the index a month or two ago.

But for the most part is only snagging the index page of new sites and ignoring new pages on existing sites.

With that said -

1. Has anyone gotten a completely new site spidered and indexed within the last 30 days? And with a PR above -0-?
2. Has anyone put new pages on an old site and had them spidered and indexed within the last 30 days?

3. If yes to either question do you have a large amount of incoming links? Or a high PR? Or, take a guess at why you think you were spidered and indexed?

It does seem there's more to this than just Googlebot being rude to new friends (especially with the G-man saying in another thread to expect daily or near daily updates).

-s-

[phish]

i added pages within the last 3 weeks to a 2 year old site with a pr6, linked from my homepage, sitemap, etc and nothing has been picked up. Googlebot crawls my entire site everyday 20-30 pages, same pages over and over, but wont get the new ones...go figure. 6 months ago it would be a day or two tops for new pages to be indexed.

-phish

[coosblues]

I was wondering the same thing. Seems like the freshbot has taken a vacation. In the past 30 days I've added about 10 new pages. Five have been spidered and only one has been indexed with a PR0. Something is definitely amiss at Google these days. A month or two ago fresh pages were in the index within 2 to 3 days. I had felt I was alone, but seeing most others are having the same tribulations. Backlinks updated last month, PR did not. A good ole fashioned update is long overdue.

[ChrisKud5]

I am glad I am not the only one!

Index page and other sub index pages (/cat/index.asp) are being crawled, but nothing else.

New pages remain blank in PR.

I put up a link to a brand new site page that has no page rank or backlinks at all like a month ago and googlebot has still not followed the link. The link was from a PR 6 to this new page with no PR, and still nothing.

What is the deal?

[Dayo_UK]

1. No, best I have got is 12-20 pages indexed for a day or two (Like Old Fashioned Freshbot - these pages are indexed and dropped). I dont think any new pages within the last 30 days is showing Page Rank yet.

2. Older PR6 Site can get new pages indexed, by linking from Home Page.

3. PR6 sites has over 300 links. New site(s) obviously have enough links for a fresh crawl at the least.

[coosblues]

Looks to me as if the freshbot is active as ever again and quickly getting new pages indexed. The past couple of weeks seemed like Google was getting a bit stale, but I had a nice deepcrawl the day before yesterday and all the new pages were added today. Prior to that those same pages had been crawled (about 2 weeks ago) but never found their way into the index. Things are looking fresher to me today than they have been in quite some time. I smell something!

[Dayo_UK]

Confirmed, Freshbot is active again - However it is not always a good sign:-

[webmasterworld.com...]

I smell something too and it is not pleasant.

[kovacs]

I've been having a whole bunch of new domains spidered over and over again. The sites went up just under a month ago, were linked from my old indexed sites, and were spidered quickly (and heavily) after that. About a week ago the index pages were added to the index (no fresh tags), and in some cases a seemingly randomly selected page as well. They all have snippets, and they seeem to be ranking ok given that none of the other pages are showing up in the index. However I am wondering if and when the rest of the pages are going to be added - they've all been spidered, sometimes several times already. I only ever seem to be being visited by 64.* bots.

[otech]

Interesting, i got deep crawled on the 13th/14th for about 98% (about 1500 pages but i have noarchive on 40% as they are printable versions of the same pages) of my site, but new pages werent added to the index...

All the pages that were crawled were updated in the cache overnight; and it deep crawled me around the same time of the month as every other month this year (site has only been up 3 months). I have only been deep crawled 3 times.

Its deep crawling existing sites and pages, but not adding newies perhaps?

[Philarmon]

Googlebot has crawled a lot of pages in the last 2 weeks on my large dynamic driven website with some good incoming links (mostly PR5, a few of PR6 and one PR7)

But, 6 weeks ago i have launched 3 new websites and i still see only the index pages of them in the index. And my other PR4 websites are updated very slowly or not at all ...

And i am really missing the PR update - it would be good to see some rewards for the hard work :) Anyways, i am not very happy with the G at the moment.

[zgb999]

Last week we put a new site online and a few days later Google had the homepage of the site in the index.

[kovacs]

Have you had the rest of the pages spidered and indexed though? I've had the root index pages indexed, but none of the other pages, which have been spidered.

[Henley]

Put up a page on a PR4 commercial site about a forthcoming local event about 7 days ago.
Arranged a news snippet on a local small town directory/community site with a link direct to the new page.
The news snippet on the directory site was picked up by Google within 24hrs and two days later the new page on my site. Now can be found #1 for all keyphrases.
Henley

[shocko]

I put up 4 new sites less than a week ago with just a few links to them and they have all been crawled and index. They range from 300 pages to over 5,000 for 1 site.

Some sites that I put up a few months ago that used to send me a decent amount of traffic suddenly are now only sending me about 1% of the amount of traffic I was getting last week?

[Dolemite]

I had a new site go live about 2 weeks ago. It was well linked almost immediately and was well spidered in the days that followed, but so far less than 1% of the pages are indexed. All the indexed pages are subdomain index pages, plus the main index page. I.E., nothing past "/" root is indexed. I know google has some data on these unindexed pages since adsense ads are generally well-targeted.

This is definitely a step backward from the previous indexing rate/depth.

[Leonardo Mello]

(first, sorry for the mistakes in english, i don't have a good english :)

I added my website in Google about 2 months ago... And since last day 13 Google are indexing my pages (109, a good number) EVERY day: 13, 14, 15 (i hope today, 16, too).

It's a miracle? I have PR0 but in many searches my website was in first place on results.

[jbgilbert]

Somebody said:

(especially with the G-man saying in another thread to expect daily or near daily updates).

I missed that thread, but the G-man is apparently not very well informed. It should be very obvious to him, and everybody else, that the problem mentioned in this thread DOES EXIST.

Any theories on what Google is up to?
- broken?
- preparing to unleash a monster change?
- at it's physical limit?

anything else?

[Dolemite]

Any theories on what Google is up to?
- broken?
- preparing to unleash a monster change?
- at it's physical limit?

I wouldn't speculate at anything so dramatic. Spidering/indexing rates have fluctuated in the past, and Google has often had tiny bugs.

Its a significant annoyance to folks like me who have spent 2 months developing a site and the next few weeks promoting it only to watch Google sit on its hands when it comes to indexing, but I'm treating it as a temporary setback until I see evidence to the contrary.