Welcome to WebmasterWorld Guest from 54.166.214.107

Forum Moderators: open

Message Too Old, No Replies

Are all crawled pages in the *live* index?

Is FreshDeepBot putting all new pages into the live index for good?

     
11:43 pm on Jul 14, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 23, 2003
posts:163
votes: 0


So, have people noticed new content which has been crawled, but not posted to the live index?

Just trying to figure out if we're going to see a July update, or if what you see is what you get.

6:06 am on July 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts:12095
votes: 0


We have no way of knowing whatever will happen again. We can only guess, which is fruitless and will drive us crazy.

I suspect new pages are getting in if they're crawled if they have enough PR, and if they don't have enough they won't. Just a suspicion, no basis in fact.

6:36 am on July 15, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Mar 3, 2003
posts:244
votes: 0


For my keywords the results from www-cw are much different than from the other datacenters.

I'm wondering if this old data or new data. And if it has something to do with an update.

I need an update! I do not like this deep/fresh continuous update thing at all!

What about the backlinks? I guess we need an update for our backlinks. I have received a lot of links in the last weeks. They aren't there yet. So, I still have hope for an update.

7:47 am on July 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


I have a bunch of pages that have been crawled but aren't live yet, but I also have pages that have been added gradually over the last several weeks.

I'm seeing many differences between the datacenters. Some have more pages of my site, but others have newer snapshots.

I need an update! I do not like this deep/fresh continuous update thing at all!

I think we'll all love it, once we see that it's working.

9:14 am on July 15, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


Deepbot has been picking files up from me but no sign of them being catalogued yet.
9:56 am on July 15, 2003 (gmt 0)

Senior Member from ES 

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1379
votes: 0


I have had a bunch of pages that were crawled (on the 5th) and updated (around the 9th?) and are now stable and happily bringing us traffic ;]

However, our index page has a fresh tag in some serps, no fresh in others. Never noticed this before...

11:24 am on July 15, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


I just see that another 10 files have entered the index in
the past few hours. I have freshbots around all the time but I think these are due to the current deepbots. (crawlers/crawls).
11:43 am on July 15, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


It is great to find out about site: ... -qwerrew and
allinurl: ... but is there anything equivalent for external
links. That would be invaluable too.
12:34 pm on July 15, 2003 (gmt 0)

New User

10+ Year Member

joined:June 16, 2003
posts:6
votes: 0


I have a new website, freshbots has been on it for 3 days, last week, after that my web site appeared in google results. Sadly I have PR at 0, that's because backlinks haven't been calculated, yet.

What I think is that google is trying to index all pages as often as it can and later, maybe every month is calculating PR , backlinks and other stuff.

12:40 pm on July 15, 2003 (gmt 0)

Junior Member from GB 

10+ Year Member

joined:Oct 16, 2002
posts:162
votes: 0


loganoski,

I'd agree with that, that was the reason they started with the freshbot in the first place and I believe GG said it was just in testing phase a while back, this is just the next step in the plan for freshbot in my opinion, they can't go recalculating PR and backlinks every day but they want to keep more and more content fresh, which is better for us all.

Cheers,
Nigel

2:25 pm on July 15, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Apr 22, 2003
posts:250
votes: 0


For us, pages have been updated, but the cache for the index has only been updated with a refresh every few days and then it goes back to the older cached version. There still needs to be a hard update, imo.
3:21 pm on July 15, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


I was wrong about an extra ten pages being entered in the index. Sorry about that.
7:00 pm on July 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Getting freshed every single day for the last few weeks, and changes to pages are being cached straightaway. Not reverting to old versions at all.
7:25 pm on July 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 22, 2002
posts:1807
votes: 1


How high is the PR on that site G1?
7:56 pm on July 15, 2003 (gmt 0)

Full Member

10+ Year Member

joined:June 13, 2003
posts:332
votes: 0


Last month the dance started on the 15th.

Looks like there may not be a 'dance' in July at this rate.

If there ever is a dance, what is going to be the new name? =)

jd

8:20 pm on July 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


>> How high is the PR? <<

Has always been white bar 0/10.

Site at this new domain went live 2003-05-04 and first appeared in Google on 2003-06-16, at the very same time as the old domain was dropped from the listings. The old domain had noindex,follow on all pages, since late May, and all "internal" navigation of the old site was modified to point to the equivalent page on the new site instead. Old site still shows PR3 even though it is nowhere in the index.

4:37 am on July 16, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 17, 2003
posts:687
votes: 0


Hi,

Sometimes your fresh file is index but it still display the old *deep crawl index. Especially when you change your title.

SO you need to check your cache to see if the fresh bot got the files into the index.

5:32 am on July 16, 2003 (gmt 0)

New User

10+ Year Member

joined:June 12, 2003
posts:25
votes: 0


Hello,

My site is a PR 1 and is getting fresh tags daily for about 30 to 60 pages out of 250 in site. Cache for these pages is 216... Problem for me is that many of these pages are dropped from the index once the fresh tags are gone.

I am tired of pasting all new links onto my index page (as one poster suggested) just to entice deepfreshy to crawl them. I don't see much benefit in sticking new links on my index page as deepfreshy's crawling doesn't mean that new pages will stick sans fresh tags.

Marcia may be right regarding her hunch that higher PR pages may have more stickiness in the index.

12:00 pm on July 16, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:July 9, 2003
posts:405
votes: 0


How do I get freshbot to my site, I'm currently indexed, but have updated the site to try to optimise it, but i'm not being crawled?
1:49 pm on July 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


Well now I can confidently report that the number of files
indexed at my site has gone up from 232 to 256 out of 340
plus 20 graphics files. Whether this was from freshbot or deepbot (both are around) I dont know.
2:34 pm on July 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 10, 2003
posts:87
votes: 0


Does anybody doubt that there will have to be some type of a regular update just for the purpose of calculating the new PR? The iteration process just seems to complicated to do on the fly. I would think that computing anchor text would be possible on the fly, but even that may be best left to a once a month adjustment.
2:35 pm on July 16, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Hmm, I bet that everything could be done gradually. For example, the index could keep track, for each page, when its PR was last updated. Then, whenever there was time, it could find the page with the most stale PR, and recalculate it based on a database of backlinks.

But, it wouldn't suprise or bother me if PR and anchor text was done monthly. I'd rather see new pages appear in the index quickly, than to have their PR be up-to-the minute.

2:58 pm on July 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


Silly me. I am slow-witted. I checked my files and none have the date against them so I assume that the newly-indexed files came from deepbot rather than freshbot. The number indexed has now dropped from 256 to 255 but that still means twenty+ more have been indexed since this morning.
4:12 pm on July 16, 2003 (gmt 0)

Junior Member from GB 

10+ Year Member

joined:Oct 16, 2002
posts:162
votes: 0


I have pages in the index with no date with content from 4 days ago (not new pages just new content), I would not trust the date to mean they are from the deep index.

Cheers,
Nigel

4:59 pm on July 16, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 13, 2003
posts:379
votes: 0


Hi, all.

I lost my major entry page from Google's listings.
I don't know what happened to it.

It used to be the next highest PR page(PR5-after home page) or the highest ranked page for my sites with most my keywords.

It is now at the bottom of my hudreds pages on Google.

When I clicked the cached version, it only showed half of the page with Broken Html code.

Is it because there might be an interruption during the crawls?

Every other pages look normal.

How can I give a weight to this page back?
Do I have to wait a month or so?

It is causing Adsense ads problem too. it is showing public ads.

I believe Googlebot or Mediabot don't understand what this page is about.

Any help will be appreciated.

5:58 pm on July 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


My Home Page has 15 July against it so Freshbot must have taken a look. It hasnt been updated for a few weeks.

The pattern I am getting from Freshbot and Deepbot makes me doubt some of the claims being made for FreshDeepbot.
Although I have had nothing like the number of Deepbots I had around three months ago when I moved my site to a new place and had to start again.

6:21 pm on July 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 11, 2003
posts:63
votes: 0


If GoogleGuy is correct then this should be the last time we see Deepbot. But it is certainly still active.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members