Welcome to WebmasterWorld Guest from 107.20.75.63

Forum Moderators: open

Message Too Old, No Replies

Googlebot is out and crawling for upcoming index.

Googlebot out and fetching pages for upcoming index.

     
10:22 pm on Jan 3, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2001
posts:2059
votes: 0


I have Googlebot spidering sites for the next index now.

You can tell that because of the IP.

Freshbot starts with 64 or 66 if i am not wrong

And the "normal" Googlebot's ip is something like:
216.239.*.*

So if you have any last changes you want to have in the upcomming index, you better hurry.

[Edit]It should really be spidering (in title) if we are talking correct terms, sorry about that[/edit]

10:35 pm on Jan 3, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 5, 2002
posts:825
votes: 0


2003-01-03 21:41:43 216.239.46.90 - 80 GET /index.htm - 200 www.mysite.com Googlebot/2.1+(+http://www.googlebot.com/bot.html)

Completed my changes not a moment too soon ;)

12:01 am on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member chiyo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 21, 2000
posts:3170
votes: 0


I'm increasingly starting to beleive that if your new pages are linked from a freshbotted page (say your front page and such) that they will even get in the permaneant main index if they were just published a few days before the index. Maybe it also depends on the extent of the PR too and if the new pages were from the same domain.

At the moment we have a page uploaded 5 days ago still hanging in there on all the updated data servers up till now. Time will tell if its just freshbot, or if we really made it into the new perm index in a few days.

Also noticed similar in previous updates but never really followed it closely.

That MAY mean that for established freshbotted sites, the cut-off time is far less and no crazy update schedule around reindex time.

Anybody else noticing similar?

12:21 am on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 15, 2002
posts:1199
votes: 0


Yes. We uploaded a page 1 day before the last crawling stopped for the last update and it is already indexed and #1 :)

We add a new section to our site every week and it's indexed by freshbot within 2 to 3 days, and it's normally indexed permanently in the following update.

12:23 am on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
posts:3454
votes: 0


chiyo,

I sure like the sound of that! Freshbot loves my "recent additions" page, that I added last month, and just got indexed at PR5. Hopefully this will be enough to get the freshbot to follow to the new content pages later in the month.

It sure is nice when you do something for the users, and you find out that it might help you in the SERPs. :)

I will be sure to keep an eye on this with the January update.

3:23 am on Jan 4, 2003 (gmt 0)

New User

10+ Year Member

joined:Aug 22, 2002
posts:32
votes: 0


Yes, the deep crawler starts his monthly work. in the moment i see few deep crawler requests on different sites. The same as every month. Beginning slowly and the next day it will crawl faster.
3:44 am on Jan 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 2, 2002
posts:44
votes: 0


How often does Googgle do a deep crawl and how often does freshbot come around? My "old site" was crawled Dec 7 probably from some links I was setting up for the redesign whcih just launched last week. I havent seen Google in the logs since then but the site was just added to the index (I assume the old site) with a poor PR of 2. However, I notice that when I checked the Google cache, it showed the new site. I'm a bit confused. Any insight? Do I make sense? Thanks
4:04 am on Jan 4, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Jan 28, 2002
posts:240
votes: 0


Thanks LAZ... I notice about an hour ago and had to finish up several pages... I hate rushing as it leads to mistakes.
4:04 am on Jan 4, 2003 (gmt 0)

New User

10+ Year Member

joined:Nov 13, 2002
posts:30
votes: 0


Ive noticed some interesting info regarding freshbot. 64.165.53.170 visited my domain today once, requesting 66 pages out of my 600ish pages - but consumed 0kb of bandwith. 64.165.53.170 was the 11th out of 400 sites to visit my domain this month and the only domain that list's 0kb of bandwith. /shrug
4:09 am on Jan 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 2, 2002
posts:44
votes: 0


Does the Freshbot identify itself from Google as well as Googlebot?
9:54 am on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2001
posts:2059
votes: 0


kwburke
It sais Googlebot, but it comes from a different ip. (it doesn't start with 216)
10:29 am on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member powdork is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 13, 2002
posts:3346
votes: 0


Freshbot 64.68.*
Deepcrawl bot 216.239.46.*

I haven't had the same luck with fresh listings and I do follow them closely. For me they will disappear 2-3 days after freshbot's last visit consistently. I'm looking to see if the timeframe starts with the last GET for the page or if it matters whether the server returns a 304 or 200. To be safe I update regularly any new important pages as well as any pages along freshbot's route just to be sure a 200 is returned. Of course, mysite.com is smallish so I can do that.
The deep crawler has shown up and for some reason asked for my robots.txt and / twice before heading on.

11:12 am on Jan 4, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 2, 2002
posts:274
votes: 0


From my figures this bot paid me a visit earlier this week. I'll never figure this stuff out..hehe. I'll have to check to see what this critter does unless someone else knows.

Referring Domain : 216.239.39.100
Total Bandwidth : 349.96 KB

2:12 pm on Jan 4, 2003 (gmt 0)

Junior Member from GB 

10+ Year Member

joined:Oct 16, 2002
posts:162
votes: 0


Bot's been here too

216.239.46.140 - - [03/Jan/2003:21:28:15 +0000] "GET /robots.txt HTTP/1.0" 200 71 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Time is in GMT.

Nigel

4:42 pm on Jan 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 2, 2002
posts:44
votes: 0


Could someone help me..

Google visited our site this am at 5:36 am but only stayed for 2 seconds and visited two pages...the home page and robots.txt. Why only stay for two seconds? Is this just the freshbot? Google first ever visit to the site was Dec 7. Had never been indexed before then, but it has been completley redesigned since then and relaunced last week. Any input? Thanks.

216.239.32.0 - 216.239.63.255
CIDR: 216.239.32.0/19
NetName: GOOGLE
NetHandle: NET-216-239-32-0-1
Parent: NET-216-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
NameServer: NS3.GOOGLE.COM
NameServer: NS4.GOOGLE.COM
Comment:
RegDate: 2000-11-22
Updated: 2001-05-11

2. 216.239.46.36 Jan. 4, 2003 at 5:36 a.m. 2 seconds 2 (0.8%) 1 (1.5%

6:35 pm on Jan 4, 2003 (gmt 0)

Preferred Member from US 

10+ Year Member

joined:Nov 27, 2002
posts:410
votes: 0


This is what I have at last check

Last 10 Visitors

Time (EST) IP Address
12:58:10 PM 68-117-211-119.charterga.net (68.117.211.119)
12:58:10 PM 68-117-211-119.charterga.net (68.117.211.119)
12:57:50 PM 68-117-211-119.charterga.net (68.117.211.119)
12:57:50 PM 68-117-211-119.charterga.net (68.117.211.119)
12:45:10 PM crawl2.googlebot.com (216.239.46.**)
12:44:00 PM pool0553.cvx34-bradley.dialup.earthlink.net (216.244.8.43)
01:17:00 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:16:50 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:16:50 PM ip68-8-210-222.sd.sd.cox.net (68.8.210.222)
01:13:40 PM crawl5.googlebot.com (216.239.46.***)

6:42 pm on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 28, 2002
posts:757
votes: 0


As far as I can tell, the fresh listings are still in the index. However, there still seems to be a difference between the fresh and perminant listings, since the fresh ones are not showing up in Yahoo, whereas the non-fresh ones are.
6:49 am on Jan 6, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 31, 2002
posts:150
votes: 0


Visitor 64.68.82.18 on Mon Jan 6 06:12:10 2003 using Googlebot/2.1 (+http://www.googlebot.com/bot.html).
The remote host was: crawler10.googlebot.com
6:51 am on Jan 6, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 31, 2002
posts:150
votes: 0


^ That would be freshbot?
7:07 am on Jan 6, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 5, 2002
posts:825
votes: 0


ya, see message 12.