homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Are all crawled pages in the *live* index?
Is FreshDeepBot putting all new pages into the live index for good?

 11:43 pm on Jul 14, 2003 (gmt 0)

So, have people noticed new content which has been crawled, but not posted to the live index?

Just trying to figure out if we're going to see a July update, or if what you see is what you get.



 6:06 am on Jul 15, 2003 (gmt 0)

We have no way of knowing whatever will happen again. We can only guess, which is fruitless and will drive us crazy.

I suspect new pages are getting in if they're crawled if they have enough PR, and if they don't have enough they won't. Just a suspicion, no basis in fact.

Wired Suzanne

 6:36 am on Jul 15, 2003 (gmt 0)

For my keywords the results from www-cw are much different than from the other datacenters.

I'm wondering if this old data or new data. And if it has something to do with an update.

I need an update! I do not like this deep/fresh continuous update thing at all!

What about the backlinks? I guess we need an update for our backlinks. I have received a lot of links in the last weeks. They aren't there yet. So, I still have hope for an update.


 7:47 am on Jul 15, 2003 (gmt 0)

I have a bunch of pages that have been crawled but aren't live yet, but I also have pages that have been added gradually over the last several weeks.

I'm seeing many differences between the datacenters. Some have more pages of my site, but others have newer snapshots.

I need an update! I do not like this deep/fresh continuous update thing at all!

I think we'll all love it, once we see that it's working.


 9:14 am on Jul 15, 2003 (gmt 0)

Deepbot has been picking files up from me but no sign of them being catalogued yet.


 9:56 am on Jul 15, 2003 (gmt 0)

I have had a bunch of pages that were crawled (on the 5th) and updated (around the 9th?) and are now stable and happily bringing us traffic ;]

However, our index page has a fresh tag in some serps, no fresh in others. Never noticed this before...


 11:24 am on Jul 15, 2003 (gmt 0)

I just see that another 10 files have entered the index in
the past few hours. I have freshbots around all the time but I think these are due to the current deepbots. (crawlers/crawls).


 11:43 am on Jul 15, 2003 (gmt 0)

It is great to find out about site: ... -qwerrew and
allinurl: ... but is there anything equivalent for external
links. That would be invaluable too.


 12:34 pm on Jul 15, 2003 (gmt 0)

I have a new website, freshbots has been on it for 3 days, last week, after that my web site appeared in google results. Sadly I have PR at 0, that's because backlinks haven't been calculated, yet.

What I think is that google is trying to index all pages as often as it can and later, maybe every month is calculating PR , backlinks and other stuff.

Made In Sheffield

 12:40 pm on Jul 15, 2003 (gmt 0)


I'd agree with that, that was the reason they started with the freshbot in the first place and I believe GG said it was just in testing phase a while back, this is just the next step in the plan for freshbot in my opinion, they can't go recalculating PR and backlinks every day but they want to keep more and more content fresh, which is better for us all.



 2:25 pm on Jul 15, 2003 (gmt 0)

For us, pages have been updated, but the cache for the index has only been updated with a refresh every few days and then it goes back to the older cached version. There still needs to be a hard update, imo.


 3:21 pm on Jul 15, 2003 (gmt 0)

I was wrong about an extra ten pages being entered in the index. Sorry about that.


 7:00 pm on Jul 15, 2003 (gmt 0)

Getting freshed every single day for the last few weeks, and changes to pages are being cached straightaway. Not reverting to old versions at all.


 7:25 pm on Jul 15, 2003 (gmt 0)

How high is the PR on that site G1?


 7:56 pm on Jul 15, 2003 (gmt 0)

Last month the dance started on the 15th.

Looks like there may not be a 'dance' in July at this rate.

If there ever is a dance, what is going to be the new name? =)



 8:20 pm on Jul 15, 2003 (gmt 0)

>> How high is the PR? <<

Has always been white bar 0/10.

Site at this new domain went live 2003-05-04 and first appeared in Google on 2003-06-16, at the very same time as the old domain was dropped from the listings. The old domain had noindex,follow on all pages, since late May, and all "internal" navigation of the old site was modified to point to the equivalent page on the new site instead. Old site still shows PR3 even though it is nowhere in the index.


 4:37 am on Jul 16, 2003 (gmt 0)


Sometimes your fresh file is index but it still display the old *deep crawl index. Especially when you change your title.

SO you need to check your cache to see if the fresh bot got the files into the index.


 5:32 am on Jul 16, 2003 (gmt 0)


My site is a PR 1 and is getting fresh tags daily for about 30 to 60 pages out of 250 in site. Cache for these pages is 216... Problem for me is that many of these pages are dropped from the index once the fresh tags are gone.

I am tired of pasting all new links onto my index page (as one poster suggested) just to entice deepfreshy to crawl them. I don't see much benefit in sticking new links on my index page as deepfreshy's crawling doesn't mean that new pages will stick sans fresh tags.

Marcia may be right regarding her hunch that higher PR pages may have more stickiness in the index.


 12:00 pm on Jul 16, 2003 (gmt 0)

How do I get freshbot to my site, I'm currently indexed, but have updated the site to try to optimise it, but i'm not being crawled?


 1:49 pm on Jul 16, 2003 (gmt 0)

Well now I can confidently report that the number of files
indexed at my site has gone up from 232 to 256 out of 340
plus 20 graphics files. Whether this was from freshbot or deepbot (both are around) I dont know.

More Traffic Please

 2:34 pm on Jul 16, 2003 (gmt 0)

Does anybody doubt that there will have to be some type of a regular update just for the purpose of calculating the new PR? The iteration process just seems to complicated to do on the fly. I would think that computing anchor text would be possible on the fly, but even that may be best left to a once a month adjustment.


 2:35 pm on Jul 16, 2003 (gmt 0)

Hmm, I bet that everything could be done gradually. For example, the index could keep track, for each page, when its PR was last updated. Then, whenever there was time, it could find the page with the most stale PR, and recalculate it based on a database of backlinks.

But, it wouldn't suprise or bother me if PR and anchor text was done monthly. I'd rather see new pages appear in the index quickly, than to have their PR be up-to-the minute.


 2:58 pm on Jul 16, 2003 (gmt 0)

Silly me. I am slow-witted. I checked my files and none have the date against them so I assume that the newly-indexed files came from deepbot rather than freshbot. The number indexed has now dropped from 256 to 255 but that still means twenty+ more have been indexed since this morning.

Made In Sheffield

 4:12 pm on Jul 16, 2003 (gmt 0)

I have pages in the index with no date with content from 4 days ago (not new pages just new content), I would not trust the date to mean they are from the deep index.



 4:59 pm on Jul 16, 2003 (gmt 0)

Hi, all.

I lost my major entry page from Google's listings.
I don't know what happened to it.

It used to be the next highest PR page(PR5-after home page) or the highest ranked page for my sites with most my keywords.

It is now at the bottom of my hudreds pages on Google.

When I clicked the cached version, it only showed half of the page with Broken Html code.

Is it because there might be an interruption during the crawls?

Every other pages look normal.

How can I give a weight to this page back?
Do I have to wait a month or so?

It is causing Adsense ads problem too. it is showing public ads.

I believe Googlebot or Mediabot don't understand what this page is about.

Any help will be appreciated.


 5:58 pm on Jul 16, 2003 (gmt 0)

My Home Page has 15 July against it so Freshbot must have taken a look. It hasnt been updated for a few weeks.

The pattern I am getting from Freshbot and Deepbot makes me doubt some of the claims being made for FreshDeepbot.
Although I have had nothing like the number of Deepbots I had around three months ago when I moved my site to a new place and had to start again.


 6:21 pm on Jul 16, 2003 (gmt 0)

If GoogleGuy is correct then this should be the last time we see Deepbot. But it is certainly still active.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved