homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 38 message thread spans 2 pages: 38 ( [1] 2 > >     
No Indexing for the Last 2 Weeks On 100+ Sites
Google brandy

 1:08 pm on Feb 25, 2004 (gmt 0)

I manage a group of 100+ sites. None of them has seen the spider for atleast 2 weeks now.This is somethig strange as i've noticed them being regularly indexed for the last 9- 10 months. Few clues;

1)I'd done a I.P reshuffle for all my sites around a month back. But i dont' think this created some probelem as bot kept coming at the sites even after. Bot also registered the changes in the cache corrosponding to the changes done on the sites.

2)Barring a few sites almost the whole group was penalised during florida.

3)I've also noticed a considerable cut in the backword link from google last week. I know that this is nothing new. Google often cuts back the backword links. What was strange however was the cut back in the no of links shown with the command "+www.site.+com"

4)Also though most of my site have returned to the top position in google, the titles reflected by google are preflorida. Strangly google had registered the post florida title changes for these sites.

Any comments?



 12:42 pm on Feb 27, 2004 (gmt 0)

Hi webnewton,

Even i am experiencing similar kind of situation.The Title and Description displayed are Pre Florida.We were in top positions after Florida and Austin but after Brandy there is a huge drop in the positions.


 1:58 pm on Feb 27, 2004 (gmt 0)

My sites (3 of them) are experiencing similar things. Google has just grabbed the index page in last 2 weeks but gone no further. 1 of these sites is new, the other 2 are well established.


 2:07 pm on Feb 27, 2004 (gmt 0)

Similar issue here...

Googlebot crawling on "some" sites is not as aggressive as it used to be,

Some backlinks have not been updated since early January -- especially those coming from internal site pages,

PR updates on internal site pages have not been done since early January.

Question is: Is it by Google design or does Google have a new problem?


 5:29 pm on Feb 27, 2004 (gmt 0)

The same issue here. Seems like a big Google spider slowdown. Maybe they are having spidering problems.


 5:59 pm on Feb 27, 2004 (gmt 0)

I have seen a ton of spidering, last weekend, but those pages spidered have not been updated....


 6:08 pm on Feb 27, 2004 (gmt 0)

Also noticed significant slowdown in spidering across 6 sites with new pages added , I don't add hundreds or thousands of pages a week maybe 10 - 20 maximum each week per site but as i have a google site search box for my visitors to search mysite am starting to feel concerned if I should go back to picosearch
any ideas please GG if this will be standard crawling from now on , no problem if it is but would help to know



 6:24 pm on Feb 27, 2004 (gmt 0)

I've been spidered plenty, but have seen no update. My new pages have not been included and PR has not been given to the new pages. They were made about 2 weeks ago. I am however being spidered deeply.


 6:48 pm on Feb 27, 2004 (gmt 0)

Googlebot was spidering my site for the first two weeks of February, must have indexed all 17,000 pages. The index is the only page so far to show a fresh date.

If GoogleGuy is listening, when will the database be updated? Many of us have made changes and want to see the results. I used one area of my site to experiment with stemming and will apply to the remainder if sucessfull.

I've read others say new sites are showing up, but have not seen any new pages other than index since first of February.


 7:07 pm on Feb 27, 2004 (gmt 0)

I am getting new title changes showing in 2 days! Being crawled more often that ever before but have not had any new pages added to the index for about 6 weeks.

My keyword searches had gone haywire as well some get many get #1 hits others show nowhere at all?


 8:52 pm on Feb 27, 2004 (gmt 0)

May be the 4 billion mark is the answer to these questions. Googlebot is busy crawling new sites and increasing the number of pages instead of crawling the existing pages more often.


 9:43 pm on Feb 27, 2004 (gmt 0)

I am seeing smaller than normal googlebot activity too. About 10% of normal - oddly enough, looks like only PR5 or above pages being hit....


 10:41 pm on Feb 27, 2004 (gmt 0)

Same observations with some of our sites...

looks like only PR5 or above pages being hit

...that too...I see only the homepage showing fresh date. :-(


 11:36 pm on Feb 27, 2004 (gmt 0)

i am observing something similar...I recently redesinged the frontpage of one of my established sites, and now its showing a month old cache... freshbot (66....something)has stopped visiting the site..however the deep googlebot(216...something) is still visiting....


 12:28 am on Feb 28, 2004 (gmt 0)

Maybe they just don't have enough servers and capacity to spider all sites?


 12:36 am on Feb 28, 2004 (gmt 0)

64.68.xx.xx is the bot. 216 is now just translation isn't it?

Our site has many of its pages at PR5. They are getting crawled harder than ever. The rest are mostly PR4 and are getting looked at less often, but still being visited.

I've read a lot of posts lately about people not being crawled. Believe me, it's not a universal problem. We're getting crawled and fresh-tagged like never before.


 4:17 am on Feb 28, 2004 (gmt 0)

my site that is showing 1 month old cache now...it was getting crawled a lot harder(was showing new freshtags on about 300 pages everytime) before i redesigned the frontpage...its only after i uploaded the new design that my site isn't getting crawled anymore....its really wierd...i dont know what to do now..


 4:25 am on Feb 28, 2004 (gmt 0)

darkroom, if all else fails, you could always put the old frontpage back and see if that helps...

[edited by: Stefan at 4:38 am (utc) on Feb. 28, 2004]


 4:33 am on Feb 28, 2004 (gmt 0)

thats the only option i can see myself with..unless googleguy adresses the problem and explain how to fix it....its been a week now and no sign of any 64.
but interestingly, when i uploaded my new redesigned frontpage, i did get freshtags 2 to 3 times consecutively..but i kept on tweaking my frontpage for getting my same rankings back...instead, not only did i loose the rankings, but also lost freshbot activity...


 2:55 pm on Feb 29, 2004 (gmt 0)

An obvious reason is that Google has 1 billion pages more to crawl. This seems to have slowed down the crawling process.

Some pages still are visited daily but from what I can see the content Google is showing is not as fresh as it used to be (on a whole) and new pages take longer to be in the index.

The main question is whether this is a temporary state (until there are more ressources for crawling) or whether this will stay.

With the competition from yahoo! I am sure Google has all interest in speeding up the crawling process again to get in fresher content faster.


 10:21 pm on Feb 29, 2004 (gmt 0)

Individuals need to understand that they are not the whole universe. Google has dramatically upped both its depth of crawl and frequency of fresh dates. Half my pages get a new fresh date every single day.


 11:57 am on Mar 1, 2004 (gmt 0)

With the replies from all of you one thing is clear now. The I.P changes i did on my sites has nothing to do with the bot not updating my sites. Also;

1)Since the last 10 months or so i've seen google updateing my sites every 3 days. If i needed to change any title for some specific occasion i did the changes 3-4 days in advance. I'd changed the titles of some of my sites on 10 feb to introduce a Valentine's special offer. Unfortunatley iven't seen any updations yet.

2)All the other sites who're ranked in the SERPS along with my sites are being updated by the bot regularly.

3)My stat shows me that bot is regularly indexing my sites, indeed a deep indexing as ever before but no updations. The date script that shows below the listing of the site is missing for more than 20 days now.

Should i wait, or there is something i can do about it. Any comments from googleguy?


 12:37 pm on Mar 1, 2004 (gmt 0)

There is no hard and fast rule. We all know that. Sometimes nothing makes sense at all. My observations:

1) This has not got anything to do with any Google Algo change.
2) The quality/PR/pages etc. of the site crawled does not matter.
3) This is a global phenomenon. Many people are experiencing this.

What could be the cause:

1) Google Servers have problems again.
2) They have not been able to index/cache sites in thoss particular servers/databases.
3) Even the Google Guy is confused/amused.




 12:46 pm on Mar 1, 2004 (gmt 0)

Same observations here:

My PR 0,1,2,3 sites are not being updated and a PR 5 site is updated every day.

Since there is no PR updated happening, I am concerned about the new sites as
Low PR - slow indexing and slow indexing - low PR :)



 12:50 pm on Mar 1, 2004 (gmt 0)

An obvious reason is that Google has 1 billion pages more to crawl.

Google did not overnight add a billion pages. They had been adding those additional pages for months and months and all they did overnight was update the web page count at the bottom of their home page and send out a press release to try to steal yahoo's thunder.


 2:40 pm on Mar 1, 2004 (gmt 0)

I am facing one more weird problem.The changes we made on some of our Pages got crawled before Brandy and they were reflected.After Brandy i see that google serp's show old pages(pre austin & florida results) and the recent changes made are not at all considered.Is this a problem with google Database.How many days more will it take to reflect the changes? Most funny thing is even for our home page with page rank 6, its showing the pre Austin & pre Florida Tilte.The links that are on Home page are not crawled since Brandy and No changes are reflected.Is any one else facing similar problem?Do you have any solution to make google crawl the changes on the site.


 6:06 pm on Mar 1, 2004 (gmt 0)

"Google did not overnight add a billion pages."

You are certainly right but Google has to actively crawl 1 billion pages more now to keep them fresh...


 6:25 pm on Mar 1, 2004 (gmt 0)

darkroom! my site is showing really old page in the cache too! so old in fact, it updated the cache to my new page but has gone back to the old one again?

this has ment losin position 1 to now bein at 10!

anyone else have this prob?

is google trin too had to index new sites without keepin up-and-runnin sites updated?


 6:26 pm on Mar 1, 2004 (gmt 0)

My log analysis indicates the pages getting left out in the cold are the brand new pages.

So maybe the problem is an overload on resources. Once a new page is thrown in the serp database, it will begin requesting update resources on a frequent basis.

Anyone got a few thousand servers they can donate to the cause ;)


 7:26 pm on Mar 1, 2004 (gmt 0)

I put in a couple of new sites just before Brandy and during that weekend.

They've been bouncing all around, new Title tag, old Title tag, in, out, back and forth. Just when I think it's making forward progress, now I'm back to the beginning. Old Title tags and the newest of the sites are not in the index. The newest being the ones added after the update began.

Don't understand it and I'm tired of being patient but it does seem like a few of us are seeing the same thing.

Maybe Google needs some Pepsid to settle it's stomach.

This 38 message thread spans 2 pages: 38 ( [1] 2 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved