Deep crawl bot out?......strange?

crawlxx - not crawlerxx

[kstprod]

If I am right, then I am getting deep crawled for some reason. Any ideas? FreshBot spidered me this morning, but I was just suprised to see that the Deep Crawl Bot has also spidered me tonight.

I was told that...

crawlerXX.googlebot.com = FreshBot
crawlXX.googlebot.com = Deep Crawl

Both bots have spidered me today, so I supposed one of them has to be the Deep Crawl, right? Why would this be happening already, isn't this a bit early?

Thanks!

Karen

[Slade]

Most of my sites have already had their deep crawl this period.

Having said that, it does take a while to spider all 3 billion pages...

[kstprod]

Slade,

I already have too.....this is the second one for this month....I was deep crawled the 4th and 5th too. Any reason to be deep crawled twice?

Karen

[Slade]

Is the bot picking up the same pages she got before?

[kstprod]

Slade,

Yes, she picked up all the pages from the deep crawl starting the 4th, and then some. She has also picked up pages that I have created only a couple of days ago. Puzzles me.

Karen

[Grumpus]

I've been being crawled continuously all month. Haven't checked to see if there are duplicate pages, but as far as I can tell, it's all the same crawl on my site...

G.

[h_b_k]

I have noticed this fact too.

my site (toolbar pr 5) has been full deep crawled today for the second time this month.

the same I have seen in october.

[annej]

I'm not getting
crawlerXX.googlebot.com
crawlXX.googlebot.com

Instead I appear to have visits from
crawler11.googlebot.com
crawler12.googlebot.com
crawler13.googlebot.com

I'm trying to figure out if I have been deep crawled as Google is still links old discontinued pages when I search.

Anne

[cyndyb109]

Certain pages on our site have been re-spidered every week, some have not been spidered since August. Any comments on why Google in not picking up these pages? I can't form any pattern. Some of those spidered as late as yesterday have links from the index page and some do not. Some are product pages and some are content pages. Content was re-freshed on some very recently and yet they were not spidered. Content that has stayed the same on other pages for over 2 months was spidered. What am I missing?

Cyndy

[troels nybo nielsen]

Welcome to WW, Cyndy

> I can't form any pattern.

Not even Google is perfect.

[optisoft]

In the last two days -

crawl4.googlebot.com
crawl5.googlebot.com
crawl7.googlebot.com
crawl1.googlebot.com

- has been to my site, deep crawling.

The bot is out and about ;)

[mahlon]

it does take a while to spider all 3 billion pages...

Wow you have a big website there Slade! :o

Just kidding!

I have seen both crawlers here!

[cyndyb109]

Thanks for the welcome:

I'm getting
crawler10.googlebot.com
crawler12.googlebot.com picking up almost the same pages every day for the last three. no deep crawls

Since we are an e-commerce site and part of it is password protected with robots txt, how can this affect what gets crawled that is not protected?

[annej]

How do you tell if it is a deep crawl?

Anne

[andreasfriedrich]

/crawl\d{1,2}.googlebot.com/  # deep crawler 
/crawler\d{1,2}.googlebot.com/ # fresh bot

Andreas

[Hoople]

googlebot.com (216.239.46.xxx) - Spider/Robot
12 Nov -- 21:09:37 -- 00:00 -- /robots.txt
12 Nov -- 21:09:37 -- -- /

Nothing since then. Deep crawl was Oct 30/Nov 2