Googlebot is out and crawling for upcoming index.

Googlebot out and fetching pages for upcoming index.

[lazerzubb]

I have Googlebot spidering sites for the next index now.

You can tell that because of the IP.

Freshbot starts with 64 or 66 if i am not wrong

And the "normal" Googlebot's ip is something like:
216.239.*.*

So if you have any last changes you want to have in the upcomming index, you better hurry.

[Edit]It should really be spidering (in title) if we are talking correct terms, sorry about that[/edit]

[coconutz]

2003-01-03 21:41:43 216.239.46.90 - 80 GET /index.htm - 200 www.mysite.com Googlebot/2.1+(+http://www.googlebot.com/bot.html)

Completed my changes not a moment too soon ;)

[chiyo]

I'm increasingly starting to beleive that if your new pages are linked from a freshbotted page (say your front page and such) that they will even get in the permaneant main index if they were just published a few days before the index. Maybe it also depends on the extent of the PR too and if the new pages were from the same domain.

At the moment we have a page uploaded 5 days ago still hanging in there on all the updated data servers up till now. Time will tell if its just freshbot, or if we really made it into the new perm index in a few days.

Also noticed similar in previous updates but never really followed it closely.

That MAY mean that for established freshbotted sites, the cut-off time is far less and no crazy update schedule around reindex time.

Anybody else noticing similar?

[SlyOldDog]

Yes. We uploaded a page 1 day before the last crawling stopped for the last update and it is already indexed and #1 :)

We add a new section to our site every week and it's indexed by freshbot within 2 to 3 days, and it's normally indexed permanently in the following update.

[BigDave]

chiyo,

I sure like the sound of that! Freshbot loves my "recent additions" page, that I added last month, and just got indexed at PR5. Hopefully this will be enough to get the freshbot to follow to the new content pages later in the month.

It sure is nice when you do something for the users, and you find out that it might help you in the SERPs. :)

I will be sure to keep an eye on this with the January update.

[warmasol]

Yes, the deep crawler starts his monthly work. in the moment i see few deep crawler requests on different sites. The same as every month. Beginning slowly and the next day it will crawl faster.

[kwburke]

How often does Googgle do a deep crawl and how often does freshbot come around? My "old site" was crawled Dec 7 probably from some links I was setting up for the redesign whcih just launched last week. I havent seen Google in the logs since then but the site was just added to the index (I assume the old site) with a poor PR of 2. However, I notice that when I checked the Google cache, it showed the new site. I'm a bit confused. Any insight? Do I make sense? Thanks

[lazyz]

Thanks LAZ... I notice about an hour ago and had to finish up several pages... I hate rushing as it leads to mistakes.

[optisoft]

Ive noticed some interesting info regarding freshbot. 64.165.53.170 visited my domain today once, requesting 66 pages out of my 600ish pages - but consumed 0kb of bandwith. 64.165.53.170 was the 11th out of 400 sites to visit my domain this month and the only domain that list's 0kb of bandwith. /shrug

[kwburke]

Does the Freshbot identify itself from Google as well as Googlebot?

[lazerzubb]

kwburke
It sais Googlebot, but it comes from a different ip. (it doesn't start with 216)

[Powdork]

Freshbot 64.68.*
Deepcrawl bot 216.239.46.*

I haven't had the same luck with fresh listings and I do follow them closely. For me they will disappear 2-3 days after freshbot's last visit consistently. I'm looking to see if the timeframe starts with the last GET for the page or if it matters whether the server returns a 304 or 200. To be safe I update regularly any new important pages as well as any pages along freshbot's route just to be sure a 200 is returned. Of course, mysite.com is smallish so I can do that.
The deep crawler has shown up and for some reason asked for my robots.txt and / twice before heading on.

[Hardwood Guy]

From my figures this bot paid me a visit earlier this week. I'll never figure this stuff out..hehe. I'll have to check to see what this critter does unless someone else knows.

Referring Domain : 216.239.39.100
Total Bandwidth : 349.96 KB

[Made In Sheffield]

Bot's been here too

216.239.46.140 - - [03/Jan/2003:21:28:15 +0000] "GET /robots.txt HTTP/1.0" 200 71 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Time is in GMT.

Nigel

[kwburke]

Could someone help me..

Google visited our site this am at 5:36 am but only stayed for 2 seconds and visited two pages...the home page and robots.txt. Why only stay for two seconds? Is this just the freshbot? Google first ever visit to the site was Dec 7. Had never been indexed before then, but it has been completley redesigned since then and relaunced last week. Any input? Thanks.

216.239.32.0 - 216.239.63.255
CIDR: 216.239.32.0/19
NetName: GOOGLE
NetHandle: NET-216-239-32-0-1
Parent: NET-216-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
NameServer: NS3.GOOGLE.COM
NameServer: NS4.GOOGLE.COM
Comment:
RegDate: 2000-11-22
Updated: 2001-05-11

2. 216.239.46.36 Jan. 4, 2003 at 5:36 a.m. 2 seconds 2 (0.8%) 1 (1.5%

[Hollywood]

This is what I have at last check

Last 10 Visitors

Time (EST) IP Address
12:58:10 PM 68-117-211-119.charterga.net (68.117.211.119)
12:58:10 PM 68-117-211-119.charterga.net (68.117.211.119)
12:57:50 PM 68-117-211-119.charterga.net (68.117.211.119)
12:57:50 PM 68-117-211-119.charterga.net (68.117.211.119)
12:45:10 PM crawl2.googlebot.com (216.239.46.**)
12:44:00 PM pool0553.cvx34-bradley.dialup.earthlink.net (216.244.8.43)
01:17:00 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:16:50 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:16:50 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:13:40 PM crawl5.googlebot.com (216.239.46.***)

[freejung]

As far as I can tell, the fresh listings are still in the index. However, there still seems to be a difference between the fresh and perminant listings, since the fresh ones are not showing up in Yahoo, whereas the non-fresh ones are.

[ga_ga]

Visitor 64.68.82.18 on Mon Jan 6 06:12:10 2003 using Googlebot/2.1 (+http://www.googlebot.com/bot.html).
The remote host was: crawler10.googlebot.com

[ga_ga]

^ That would be freshbot?

[coconutz]

ya, see message 12.