Google Deepish Crawl is on

For those whose idea of sport is tail -f /path/to/log ¦ grep googlebot

[Clark]

I do believe the Googlebot is doing a deepish crawl right now for those who like to follow it.

[mcavic]

cat access_log ¦ grep googlebot ¦ cut -d"[" -f 2 ¦ cut -d":" -f 1 ¦ uniq -c

Nice, thanks!

[xlcus]

cat access_log ¦ grep googlebot ¦ cut -d"[" -f 2 ¦ cut -d":" -f 1 ¦ uniq -c

Nice, thanks!

And if you want to count GoogleBot visits per month...

cat access_log ¦ grep googlebot ¦ cut -d"[" -f 2 ¦ cut -d"/" -f 2- ¦ cut -d":" -f 1 ¦ uniq -c

(Probably pretty obvious to most people, but if you want to count all visits instead of just GoogleBot, simply remove the

[1][b] "grep googlebot ¦ " [/b][/1]

bit)

[Critter]

Actually, to be more accurate you'd want to exclude the adsense bot like this:

cat access_log ¦ grep -i googlebot ¦ grep -vi mediapartners ¦ cut -d"[" -f 2 ¦ cut -d":" -f 1 ¦ uniq -c

Peter

[cyberprosper]

grep -c Googlebot access_log

This will give you the number of visits by googlebot. This will not count the adsense googlebot...

[Clark]

BTW, the deep crawl is still very much on. So far 10,000 pages last weekend, 10,000 pages this weekend.

[galaga]

I checked my webstats and googlebot hasn't hit my site since April. At least thats the last time I saw crawlerxx.googlebot.com. I checked and none of the ip addresses showing relate to googlebot. Has Google abandoned me because I don't update my pages much anymore?

[bobosse]

well, I am luckier than you are :-)
So far over 3,000 pages today, 4th day in a row!

Bobosse

[johnnydequino]

Yesterday I had all my late june/july changes on my home page cached by google, but none of the new subpages Today, it's back to the old cache on my home page. Is this normal?

jd

[mipapage]

johnnydequino,

1st rule of Google: There is no normal.

Is this normal?

Yes. ;-]

[g1smd]

Site online at the beginning of May, appeared in the index in mid-June, stable since then. Been freshed nearly every day for the last few weeks. Still showing no PR. Site is #1. Cache is always up to date, currently with 13 July 2003 date and showing changes made on the 12th July.

[teeceo]

I haven't seen this bot before: "203.27.237.122 - - [14/Jul/2003:12:10:54 -0700" is this the new deepbot or something else?

teeceo.

[bolitto]

I would like to second teeceo's question.

What is this IP doing? Is it real Google or what?

[g1smd]

Connection refulsed if you try that IP address.

What does a real Google IP do if you try one?

[mcavic]

It doesn't look like Google to me. According to APNIC, 203.27.237.x belongs to:

On Australia Pty Limited
157 Walker St
North Sydney
NSW 2060
AU

[avalon]

I put in place 301 redirects to my new domain in mid May and am also showing a PR#1 with freshbot visits almost every day. Pitty about the low PR because my old domain had a PR#5. Google help advised me there is no penalty and to just sit tight.

I had thought the deepbot IP was 216.239.46 but I havn't seen him at all despite 1,700 indexed pages.

The back links are one of my main concerns, I thought google directory would be in step with dmoz but it doesn't seem to have been updated for a couple of months. It also hasn't picked up my Yahoo listing yet.

I've just checked my logs and I havn't seen 203.27.237.122 at all.

[bolitto]

Well isn't that great.

After the May fiasco some people call an update all we need now is a bunch of fake Googlebots.

[acronym]

I'm seeing a lot of activity from GoogleBot:

EX030701.LOG: 6111
EX030702.LOG: 7967
EX030703.LOG: 7043
EX030704.LOG: 3792
EX030705.LOG: 2646
EX030706.LOG: 4156
EX030707.LOG: 8725
EX030708.LOG: 10020
EX030709.LOG: 8039
EX030710.LOG: 7773
EX030711.LOG: 7151
EX030712.LOG: 2682
EX030713.LOG: 4159

But since they stopped using the deep bot IPs, I don't understand how to determine whether this activity is fresh or deep. Can anyone explain this?

P.S. I used the "DOS" FIND command to get these counts:

find /c "googlebot" *.log >google.txt

[Clark]

Yup, kinda deepish crawl every day now...slowly some of it is appearing here and there..