homepage Welcome to WebmasterWorld Guest from 54.161.214.221
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Googlebot is out and crawling for upcoming index.
Googlebot out and fetching pages for upcoming index.
lazerzubb




msg:81990
 10:22 pm on Jan 3, 2003 (gmt 0)

I have Googlebot spidering sites for the next index now.

You can tell that because of the IP.

Freshbot starts with 64 or 66 if i am not wrong

And the "normal" Googlebot's ip is something like:
216.239.*.*

So if you have any last changes you want to have in the upcomming index, you better hurry.

[Edit]It should really be spidering (in title) if we are talking correct terms, sorry about that[/edit]

 

coconutz




msg:81991
 10:35 pm on Jan 3, 2003 (gmt 0)

2003-01-03 21:41:43 216.239.46.90 - 80 GET /index.htm - 200 www.mysite.com Googlebot/2.1+(+http://www.googlebot.com/bot.html)

Completed my changes not a moment too soon ;)

chiyo




msg:81992
 12:01 am on Jan 4, 2003 (gmt 0)

I'm increasingly starting to beleive that if your new pages are linked from a freshbotted page (say your front page and such) that they will even get in the permaneant main index if they were just published a few days before the index. Maybe it also depends on the extent of the PR too and if the new pages were from the same domain.

At the moment we have a page uploaded 5 days ago still hanging in there on all the updated data servers up till now. Time will tell if its just freshbot, or if we really made it into the new perm index in a few days.

Also noticed similar in previous updates but never really followed it closely.

That MAY mean that for established freshbotted sites, the cut-off time is far less and no crazy update schedule around reindex time.

Anybody else noticing similar?

SlyOldDog




msg:81993
 12:21 am on Jan 4, 2003 (gmt 0)

Yes. We uploaded a page 1 day before the last crawling stopped for the last update and it is already indexed and #1 :)

We add a new section to our site every week and it's indexed by freshbot within 2 to 3 days, and it's normally indexed permanently in the following update.

BigDave




msg:81994
 12:23 am on Jan 4, 2003 (gmt 0)

chiyo,

I sure like the sound of that! Freshbot loves my "recent additions" page, that I added last month, and just got indexed at PR5. Hopefully this will be enough to get the freshbot to follow to the new content pages later in the month.

It sure is nice when you do something for the users, and you find out that it might help you in the SERPs. :)

I will be sure to keep an eye on this with the January update.

warmasol




msg:81995
 3:23 am on Jan 4, 2003 (gmt 0)

Yes, the deep crawler starts his monthly work. in the moment i see few deep crawler requests on different sites. The same as every month. Beginning slowly and the next day it will crawl faster.

kwburke




msg:81996
 3:44 am on Jan 4, 2003 (gmt 0)

How often does Googgle do a deep crawl and how often does freshbot come around? My "old site" was crawled Dec 7 probably from some links I was setting up for the redesign whcih just launched last week. I havent seen Google in the logs since then but the site was just added to the index (I assume the old site) with a poor PR of 2. However, I notice that when I checked the Google cache, it showed the new site. I'm a bit confused. Any insight? Do I make sense? Thanks

lazyz




msg:81997
 4:04 am on Jan 4, 2003 (gmt 0)

Thanks LAZ... I notice about an hour ago and had to finish up several pages... I hate rushing as it leads to mistakes.

optisoft




msg:81998
 4:04 am on Jan 4, 2003 (gmt 0)

Ive noticed some interesting info regarding freshbot. 64.165.53.170 visited my domain today once, requesting 66 pages out of my 600ish pages - but consumed 0kb of bandwith. 64.165.53.170 was the 11th out of 400 sites to visit my domain this month and the only domain that list's 0kb of bandwith. /shrug

kwburke




msg:81999
 4:09 am on Jan 4, 2003 (gmt 0)

Does the Freshbot identify itself from Google as well as Googlebot?

lazerzubb




msg:82000
 9:54 am on Jan 4, 2003 (gmt 0)

kwburke
It sais Googlebot, but it comes from a different ip. (it doesn't start with 216)

Powdork




msg:82001
 10:29 am on Jan 4, 2003 (gmt 0)

Freshbot 64.68.*
Deepcrawl bot 216.239.46.*

I haven't had the same luck with fresh listings and I do follow them closely. For me they will disappear 2-3 days after freshbot's last visit consistently. I'm looking to see if the timeframe starts with the last GET for the page or if it matters whether the server returns a 304 or 200. To be safe I update regularly any new important pages as well as any pages along freshbot's route just to be sure a 200 is returned. Of course, mysite.com is smallish so I can do that.
The deep crawler has shown up and for some reason asked for my robots.txt and / twice before heading on.

Hardwood Guy




msg:82002
 11:12 am on Jan 4, 2003 (gmt 0)

From my figures this bot paid me a visit earlier this week. I'll never figure this stuff out..hehe. I'll have to check to see what this critter does unless someone else knows.

Referring Domain : 216.239.39.100
Total Bandwidth : 349.96 KB

Made In Sheffield




msg:82003
 2:12 pm on Jan 4, 2003 (gmt 0)

Bot's been here too

216.239.46.140 - - [03/Jan/2003:21:28:15 +0000] "GET /robots.txt HTTP/1.0" 200 71 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Time is in GMT.

Nigel

kwburke




msg:82004
 4:42 pm on Jan 4, 2003 (gmt 0)

Could someone help me..

Google visited our site this am at 5:36 am but only stayed for 2 seconds and visited two pages...the home page and robots.txt. Why only stay for two seconds? Is this just the freshbot? Google first ever visit to the site was Dec 7. Had never been indexed before then, but it has been completley redesigned since then and relaunced last week. Any input? Thanks.

216.239.32.0 - 216.239.63.255
CIDR: 216.239.32.0/19
NetName: GOOGLE
NetHandle: NET-216-239-32-0-1
Parent: NET-216-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
NameServer: NS3.GOOGLE.COM
NameServer: NS4.GOOGLE.COM
Comment:
RegDate: 2000-11-22
Updated: 2001-05-11

2. 216.239.46.36 Jan. 4, 2003 at 5:36 a.m. 2 seconds 2 (0.8%) 1 (1.5%

Hollywood




msg:82005
 6:35 pm on Jan 4, 2003 (gmt 0)

This is what I have at last check

Last 10 Visitors

Time (EST) IP Address
12:58:10 PM 68-117-211-119.charterga.net (68.117.211.119)
12:58:10 PM 68-117-211-119.charterga.net (68.117.211.119)
12:57:50 PM 68-117-211-119.charterga.net (68.117.211.119)
12:57:50 PM 68-117-211-119.charterga.net (68.117.211.119)
12:45:10 PM crawl2.googlebot.com (216.239.46.**)
12:44:00 PM pool0553.cvx34-bradley.dialup.earthlink.net (216.244.8.43)
01:17:00 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:16:50 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:16:50 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:13:40 PM crawl5.googlebot.com (216.239.46.***)

freejung




msg:82006
 6:42 pm on Jan 4, 2003 (gmt 0)

As far as I can tell, the fresh listings are still in the index. However, there still seems to be a difference between the fresh and perminant listings, since the fresh ones are not showing up in Yahoo, whereas the non-fresh ones are.

ga_ga




msg:82007
 6:49 am on Jan 6, 2003 (gmt 0)

Visitor 64.68.82.18 on Mon Jan 6 06:12:10 2003 using Googlebot/2.1 (+http://www.googlebot.com/bot.html).
The remote host was: crawler10.googlebot.com

ga_ga




msg:82008
 6:51 am on Jan 6, 2003 (gmt 0)

^ That would be freshbot?

coconutz




msg:82009
 7:07 am on Jan 6, 2003 (gmt 0)

ya, see message 12.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved