Forum Moderators: open

Message Too Old, No Replies

Has the deep crawl started in advance of the update?

Yes, it's the deep crawlers that are visiting

         

uber_boy

1:38 pm on Apr 10, 2003 (gmt 0)

10+ Year Member



Given that I've seen nothing here about an update -- other than the usual soul-wrenching anxiety that precedes an update -- I was more than a bit shocked this morning to discover that my site was being punded by the Deep Crawlers. A quick glance reveals that Crawl10, Crawl11, Crawl12, Crawl13, and Crawl16 have all visited in the last few minutes. Anyone having a similar experience or know what's up?

bobmark

2:53 am on Apr 11, 2003 (gmt 0)

10+ Year Member



There was some discussion of this last month.
There has always been a lot of variation in how many pages freshbot took from a site - some think it is related to PR or other factors.
However, several people noted a lot deeper crawling by freshbot IP's in the 64 range last month than they had previously experienced. In my case the typical 30-50 became well over 100, but I still had a full deepcrawl by 216.x at the normal time.
The past few days I have also had a ton of freshbot hits.
It may be that Google is doing a better job of - or experimenting with - co-ordinating the 64.x and 216.x crawl results into their database. In the past, freshbot results seemed to be totally wiped out by the update, which never seemed very efficient. I mean, I understand the point of "daily fresh listings" but why delete and re-crawl pages that have not changed since freshbot initially found them?
I haven't analysed my serps for certain pages in detail, or compared the freshbot and deepbot logs entries to see if 216. skips some pages that were recently crawled by 64., but I have a suspicion that Google is starting to use the two bots in concert to update the database. (it would be interesting to see if 304's appear in the 216.x log entry for pages recently crawled by 64.x).
This is based on pages seemingly lodged in the database that were added after the last deep crawl, whereas before the freshbot results seemed to disappear from the index in hours or days at the most.

Also, I echo mfishy and skipfactor - AXS is great for instant log checking. I don't use it on all pages just enough to let me know what's going on at any moment.

uber_boy

2:59 am on Apr 11, 2003 (gmt 0)

10+ Year Member



Thanks for weighing in on this issue, Bobmark. But tell me: why do you attach so much weight to the IP address and no weight at all to the domain names that Google assigns to its bots? Google is staffed by some of the best and brightest and surely it's safe to assume that there is some method behind their differentiation of bots into crawl## and crawlER##? If so, I think it is safe to assume that the Deep Crawl is underway as I've now served up more than 40,000 pages to the so-called Fresh Bots of 64 (who happen to have domain names of the crawl## variety) in less than 12 hours.

crobb305

5:10 am on Apr 11, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



For all you poor FOOLs who are waiting on a update some time soon, forget about it. We are back on end of the month updates and "yes" deepbot is out rightnow doing the monthly crawl so.......

The update has started. LOL

[webmasterworld.com ]

This 33 message thread spans 2 pages: 33