deep crawling?

[icebane]

Has anyone been experience a lack of deep crawling the past week?

Thanks for any info.

[usavetele]

Maybe Googlebot farted and died from the stink!

[Stefan]

D'accord, rfgdxm1. I had what looked like an entire deepcrawl of every page on the site a few days ago, (about 190 pages), and on most other days, lately, the bot is visiting all the main pages. Last PR update the index went to 6, and the main pages are 5, so that might help.

It was nice seeing some of those -in changes move through yesterday... it moved us up on some of the search terms that the .coms target. We're still #1 on our non-com kw's.

Google likes true .org's. No doubt about it.

[MS_Excel]

Has anyone been experience a lack of deep crawling the past week?

Just the opposite for me. I have even been considering a robot file to try and save some bandwidth!

[BroadProspect]

I must WARN peple, what we are seeing in crawler alert is that many people use a "agent tag" spoofing and setting it to googlebot on their offline complete site download, so many of you that may think googleBot visited them may have been visited by one of their comptitors who tries to check them out using the googleBot agent signature, the best thig is beside the agent tag to see that the IP matches the google datacenters ranges, there seems to be a real increase in that technique, I would say that about 13% of the people who use the crawler alert service (and we are talking many '00 of sites around the world, since it is a free service) have been identified to be scanned by their compitorors using a spoofed "GoggleBot agent tag" requests.
hope this helps
/BP

[coconutz]

Freshbot daily, occasional multiple visits and deeper pages requested about every three days. Lots of Googlebot activity on an ecom site.

2003-12-24 04:23:40 64.68.82.18 - GET /page.htm - 200 www.mysite.com Googlebot/2.1+(+http://www.googlebot.com/bot.html)

[BroadProspect]

OK people, It seems that this is it, now we are seeing googlebot start on a few '000 out of the total sites we monitor with crawler alert, all deep crawl.
let the party begin and let's deep rock & crawl roll :->
/BP

[MS_Excel]

BroadProspect, many high PR sites(millions) are deep crawled each day every day.

[BroadProspect]

we know the profile of the sites we are monitoring and what we dectect is change in trend , so if so many sites were not crawled for quite a while and suddenly they all start ..
/BP

[24bit]

Wow, just the opposite for me. In the last few days, I've had 2580 hits from the bot on a website with only about 200 pages.

[MS_Excel]

BroadProspect, I'm not saying it means nothing. Does the same "thing" happen after each deep crawl on thes sites? If so, what is the "thing" that happens.

[BroadProspect]

the "thing" is crawling all the pages of the site in alsmot the same order and crawling the all, deep.
This is what we identify as deep crawl
/BP

[MS_Excel]

Actually want I mean is why do you say "let the party begin and let's deep rock & crawl roll" I got the impression that it signifies something for you?

[baron13]

Thats really strange: In my business I always see that google is only switching between different datacenters...nothing more happens! But 2 of the sites at the top of the serps get freshtags every second day! But really only these 2 sites! All other sites don't get freshtags! This 2 sites get permanent new freshtags since 2 weeks!

I don't know what I should think about that....

[BroadProspect]

after the deep crawl we expect the re-calculation of offline peneltics for over optimized sites, so for me the all over optimization filter is like playing "chiecken", you know the game where you drive fast to the edge of the clief but should'nt go beyond it, we would like to see if the penelaized sites are still "too optimized" or managed to stop right before falling of the clief :->
Also the "dance" as it calls for the new PR calculation and SERP are done AFTER the deep crawl usually ..
/BP

[exmoorbeast]

Hey Broad prospect... that is a nice service you run there. Any chance of sending those inquisitive bots this way!

[peterdaly]

I agree with BroadProspect, the deep crawl has started. A few thousand pages of mine have already been gobbled up.

BroadProspect, do you know if it is ramping up faster than usual? Seems faster than usual to me, but I'm closest attention to a new site that's never had anything but a fresh crawl before.

[BroadProspect]

starnge, the speed of the deep crawl is slower then ever and it becoming slower ans slower within each passing hour.
it seems like they started and then decided to stop slowly, just like they grabbing new limited test data, this is the 1st time I am experiencing this.
/BP

[peterdaly]

starnge, the speed of the deep crawl is slower then ever and it becoming slower ans slower within each passing hour.

I spoke too soon, and just watched the same thing. That being said, more pages were skarfed up over the first 8 hours or so than I remember seeing recently.

Seems to me as well like it's currently not crawling.

[BroadProspect]

>> Hey Broad prospect... that is a nice service you run there.

thanks exmoorbeast, it is not really just us, we just provide this free service, it is all about the '000 of webmaster around the world using it and by that contributing to the general knowlage about how the robots (and not just googleBot) are acting.

We are thinking also puting on the site a daily report of each robot crawling activity per geographical region and ip range so all of the webmasters who use the service will know not just the crawling activity on their site but on all the WWW .

BTW, the crawler alert service existence was spread by a word of mouth but this thread brought something like 500 new webmaster to use the service :->

Thanks again to everyone! This has always been the best forum (2nd just to the nights at the pub conferences after hours ;-) where you really here the intresting stuff ... )

/BP

[JasonHamilton]

One of my new sites has gotten 1350 page views by googlebot in the last 24 hours.

[Chico_Loco]

Has anyone been getting further visits from GoogleBot in the past 2 or 3 days?

Other than the MediaPartners Bot I have seen nothing, which is weird because it usually sucks down between 30 & 100 pages daily.

[BroadProspect]

no, very very quite .... EVERYWHER!
/BP

[GoogleGuy]

I think the bots took a break on Christmas day to chow down on some turkey or peanut butter balls or something, but about an hour ago I saw many of the bots groan, sit up, and roll out the door. Maybe they had too much candy or something, but it looks like they're out the door and back on their regular http diet now. There's still one bot lying here that was doing shots of amaretto last night; I'll nudge it a little and see if I can wake it up.

[Chico_Loco]

ha ha ha ha ha.....

Now, I thought computers didn't need a break... wasn't that one of the reasons they got so popular :):)

Can't believe you're on this late... of course you're on western time.. poor old me is on eastern time :P

Happy Holidays GG!

[nileshkurhade]

GoogleGuy,

Thanks to your advise my site is back on top.

[wanna_learn]

I love GG for the way he wraps around the MEANING of his statement!

[div01]

Hmmm Amaretto...I feel like a Dr. Pepper, or two.

[Stefan]

I had precisely one file taken on Dec 25, the index.htm, and surprisingly, there was no robots.txt request preceding it. For Dec 26, UTC, I had precisely zero 64.68.xx.xx visits, a rare thing.

The bots got a couple of days off at Christmas, eh? Cool. I hope the little dears are all rested and ready to go... I just put 16 pages online today with lots of nice content.

(I wish I didn't just get my log files zipped once a day... bet it's been all over the site the last 25 hours.)

[uncle_bob]

Seems the googlebots are back out and about. Just checked the daily logs, and the bots went deep! They must need the exercise to work of all that Christmas turkey.

[GodLikeLotus]

Has anyone been experience a lack of deep crawling the past week?

For the past 3 weeks thanks to my hosting jokers. Several periods of downtime and a robot.txt file that returned a 403 error message "forbidden" after working fine, just changed thanks to the host doing something.

Googlebot, feel free to visit anytime, we have even improved the sitemap just for you.