Has the deep crawl started in advance of the update?

Yes, it's the deep crawlers that are visiting

[uber_boy]

Given that I've seen nothing here about an update -- other than the usual soul-wrenching anxiety that precedes an update -- I was more than a bit shocked this morning to discover that my site was being punded by the Deep Crawlers. A quick glance reveals that Crawl10, Crawl11, Crawl12, Crawl13, and Crawl16 have all visited in the last few minutes. Anyone having a similar experience or know what's up?

[skipfactor]

Only 64.* here this morning, you sure it's 216.*?

[uber_boy]

My logfiles don't record the IPs for numbers that resolve, so I can't answer that. However, past experience has shown that the 216s have domain names of the crawl## variety whereas the 64s take the form of crawler##. If this is the case, then it's definitely the 216s that are hammering me at the moment...

[DotBum]

Didn't Brett say something about them switching IP addresses last month? Something about deepcrawl bots doing freshbot crawls & vice versa

[creative craig]

Already talking about this one :)

[webmasterworld.com...]

Craig

[mfishy]

I have seen no sign of the deepcrawl. People keep mentioning it hitting their pages.

Can anyone confirm that they have had recent visits from Googlebot in the 216 RANGE?

[nadsab]

I can definitely say that I was deep crawled. I have reverse DNS lookups on my stats package. It was google.

[alxdean]

Argh!
I was one of the few mad people that was silently happy that the update had not started yet, as it would give me more time to finish my site redesign and get the new site indexed by google before June. I guess I'll be doing an allnighter once again.
back to work! back to work! no time! no time!

[skipfactor]

Can anyone confirm that they have had recent visits from Googlebot in the 216 RANGE?

Underscore "216 RANGE", forget the crawler & reverse DNS. Anyone seen 216.* CONFIRMED?

[yankee]

No 216 ip's here. All 64 ip's. The deep crawl HAS NOT started, that's bad information. It usually starts 4 or 5 days after the dance begins.

[hetzeld]

Haven't seen any 216 recently... but been hit more than 100 times by 64.** this morning.

Dan

[bobmark]

No one has yet posted a definitive "yes":

"I can definitely say that I was deep crawled. I have reverse DNS lookups on my stats package. It was google."

but google from what IP?
Has anyone seen a bot from 216.x?
I have a ton of 64.x freshbot hits in the past 24 hours but no 216.x's

[uber_boy]

Okay, so I'm embarrassed. I've done some digging and discovered that, despite the crawl## domain names, it's the 64s that are hammering my site. My apologies to all whose heart may have skipped a beat because of my initial posting. But that said, if Google's altered its naming conventions, how sure can we be that its not started messing with its IPs as well?

[creative craig]

If you have a read of the thread that I posted, in it Brett said that back in 2001 it was a normal thing to be deep crawled before an update :)

Craig

[uber_boy]

I did read that thread, Craig, but what I'm now confused about is whether I am indeed being deep crawled. As noted, the IP addresses are of the 64.* variety, whereas the domain names are of the crawl## variety. This is a combination I've never encountered before and, thus, I'm not sure what to make of it...

[skipfactor]

No worries Uber, 1 less cup of coffee needed today. :)

Sounds like you could use some better log analysis software. I use AXS over the fancy site stats my host provides:

[webmasterworld.com...]

[taxpod]

Nothing deep here either but plenty fresh.

[mfishy]

This is what I suspected-plenty of fresh bot stuff but no deepcrawler.

BTW, if your stats/tracking do not even show ips you may want to consider getting a script that is adequate.

The talk of deepcrawl has become like the update talks. It's like when someone cries out that the update has started because their pages positions have changed but there is no change in backlinks.

Just to be on the same page let's consider for now the deepcraler to be of the 216 Range.

Unlike the update, the deepcrawl is actually important to webmasters- in terms of actions that are required.

[uber_boy]

I know I should just let go of this thread, but I'm still a bit worried that everyone here may have been relying a little too much on the 216 vs 64 distinction. I've already noted that the 64s hammering my site don't have the crawler## domain names that other 64s have had but, rather, the crawl## sort of domain name of the 216s. Another interesting thing to note is that, in the past, when the 64s visited, they'd read about 1000 pages/hour. In contrast, the 64s currently visiting me are reading in 3000 pages/hour. Can anyone sort this out for me?

[Gibble]

I agree completely, I'm more worried about being crawled so I know I'll have my latest content in the database, than I am about the update, as long as it's been crawled I know it will go through the dance eventually...plus the fact I missed the last crawl...stupid robots.txt file (still mad at myself :p)

[nadsab]

Yes,

It's 64.x for me too. Fifty pages, but 64.x.

Fresh Bot does that many pages at once? What's the diff then?

[taxpod]

50 is not a lot of pages for freshy. She's pulled 5,000 for me today. Typically she takes more than a thousand a day from me except when she sleeps in.

Can the mods please change the title of the thread to deepcrawl false alarm or something along those lines?

[darkroom]

hmm...still waitin for the Google Update

Google Guy...can you give us an update when will the google update take place please? if not..then any hint please? lol

[Camster]

Yesterday (latest stats I can see), I had several hundred hits from the 216.* googlebots compared with several thousand from the freshbot ips.

However, there's also a page on our site that triggers errors from googlebot and emails me with the user agent. I get a couple notification emails here and there throughout the month and a burst of them during the deep crawls. Got 15 (ie a burst) of these messages between 4a and 10a CDT today.

Seems deeper than fresh to me, based on this unscientific analysis...

[snowfishin]

300 pages in last 24 from the fresh no sign of the deep crawler

[uber_boy]

For what it's worth, I'm conviced that the Deep Crawl has started. As for why I'm saying this, you can get some idea by visiting:

[webmasterworld.com...]

To what I've said there I would add that Google has been hammering my site for five hours now and has read more than 15,000 pages in the process...

[Alternative Future]

I have been pinging the www. www2. www3. servers for a few days now and always had the IP 216.239.33.100 returned for both the www2. & www3. servers until my recent ping reported back with 216.239.33.104!

Was wondering if this was a regular thing with the servers?

-gs

[uber_boy]

I've served up more than 30,000 pages to Google so far today, all to bots having an IP address in 64.* range, but bots whose domain names are of the crawl## variety (as opposed to the crawlER## variety). Why am I the only one that thinks the Deep Crawl is underway?

[teeceo]

For all you poor FOOLs who are waiting on a update some time soon, forget about it. We are back on end of the month updates and "yes" deepbot is out rightnow doing the monthly crawl so.......

teeceo.

[why2kit]

Well I guess I'm just a fool who is waiting, but according to [y2kinternet.com...] it's just freshbot.

Of course I could just be a stupid amateur.