Welcome to WebmasterWorld Guest from 54.167.29.254

Forum Moderators: open

Message Too Old, No Replies

Googlebot is back!

It looks like the deep crawl is starting.

     
9:28 am on Mar 9, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 9, 2002
posts:231
votes: 0


Google started to deep crawl my site in the last 15 minutes. Anyone else notice this?
9:35 am on Mar 9, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 31, 2002
posts:150
votes: 0


Yes I'm being invaded too. It's the freshbot here though, on 2 sites. Very hungry, must have had nothing to eat for a few days..

Crawler 11 on 64.68.82.xx

9:43 am on Mar 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 1, 2002
posts:1580
votes: 0


I haven't seen the deep crawl bot yet, but to start crawling this fast (within 60 hours of the last update) is a complete nonsense.

Altavista....refreshed within 48 hours, Google.....Refreshed within the last 6 weeks if you are lucky!

Let's start a thread about the next update...sometime in April if Google can recalculate its PR that early.

Google if you can't calculate PR faster than every 6 weeks, stop putting so much importance on something that is way out of date.

10:20 am on Mar 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 5, 2001
posts:2728
votes: 8


I was just thinking what is being talked about here, I haven't seen googlebot to my site in a few days then I look at my network connectivity logs and see tons of googlebots and netscape looking bots connected to it non-stop so guess what, he's bacckkkkkkkkkkkkkkkkkkk.
10:55 am on Mar 9, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 21, 2002
posts:74
votes: 0


At my place I still saw old results on www 2 hours ago ...
So the update is probably not finished yet :)
But I don't see googlebot an ony of my sites ...
11:46 am on Mar 9, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 8, 2003
posts:81
votes: 0


Freshbot was at my site on Friday, took a break on Saturday, and is back today. No sign of the deepcrawler yet.
11:51 am on Mar 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 2, 2003
posts:724
votes: 0

Googlebot/2.1 ( http://www.googlebot.com/bot.html) stopped by this morning. No breakdown of whether its 216* or 64*... Is this FreshBot?
1:35 pm on Mar 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


I've been crawled every day for weeks. The old truism that Freshbot disappears during the update is false.
2:08 pm on Mar 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 17, 2003
posts:687
votes: 0


> Very hungry, must have had nothing to eat for a few days..

LOL! same here! :)

3:22 pm on Mar 9, 2003 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


Google if you can't calculate PR faster than every 6 weeks, stop putting so much importance on something that is way out of date.

PR is only one of many factors that Google uses in determining search placement, so there's no need for it to be recalculated in real time. Don't confuse the desires of Webmasters with the needs of users. :-)

3:56 pm on Mar 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member rfgdxm1 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 12, 2002
posts:4479
votes: 0


>PR is only one of many factors that Google uses in determining search placement, so there's no need for it to be recalculated in real time. Don't confuse the desires of Webmasters with the needs of users. :-)

Right. For the average searcher, if Google is using a 6 week old database this isn't a real problem.

4:02 pm on Mar 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 2, 2003
posts:724
votes: 0


I'm still not completely clear on how much influence FreshBot has with Deep Crawl when it comes to generating the index every month - But I do know that Freshbot is in there almost every day updating our news sites and making it available to the world Looong before the index is completed.
5:10 pm on Mar 9, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 31, 2002
posts:148
votes: 0


Net Guy or anyone, does that mean if you get visits from freshbot, and you made changes to your title tag, you wouldn't have to wait for the next update to see the changes?

Can your position in the serps move based on this or do you have to wait for the dance to see any movement.

Thanks.

5:25 pm on Mar 9, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 14, 2002
posts:41
votes: 0


Buckley, I have experienced this recently. I was changing a page and noticed that the changes were reflected in Google twice in 1 week.
10:29 pm on Mar 9, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 9, 2002
posts:231
votes: 0


Must have been the freshbot... I see ~50 or so new pages in the index today, just hrs after the crawl. I am expecting some 20,000 dynamic pages to be indexed in the next deep crawl. I have my fingers crossed.
10:35 pm on Mar 9, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 22, 2002
posts:519
votes: 0


Hi, sorry for newbie question! But, how exactly do we see if Googlebot is at our site?
6:02 pm on Mar 10, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 5, 2003
posts:807
votes: 0


freshBot: 64.68.82.* bah, listed for only a few days, short dinner date then it dumps you.
deepcrawler: 216.239.46.* Good bot, it likes you, listed until death do you depart. So don't make him mad or it will divorce your site. You can divorce the Googlebot by using your robot.txt file. It's much cheaper and faster than going to the courts.
6:38 pm on Mar 10, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Buckley,

Yes to both questions. Our fearless WebmasterWorld leader calls this "surfing the fresh wave."

Jim

7:03 pm on Mar 10, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 30, 2002
posts:437
votes: 0


Over 3000 hits from GoogleBot so far today.

They are all Freshbot. No need to get alarmed yet about the deep crawl. Not based on information from my server at least.

-Pete

7:17 pm on Mar 10, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 2, 2002
posts:374
votes: 0


I'm wondering if there is a change underway.
I don't know if it is just a change in how Google handles my site or something more.
In the past, freshbot would take my index and main directory pages but nothing else. The past 2 days it is deep crawling the entire site.
Is this a pattern others have experienced in previous months or is it possible that 64.bot is doing 239.bot's work this month?
Ideas?
8:44 pm on Mar 10, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 15, 2000
posts:482
votes: 0


now and then the freshbot crawled very deep on our sites.
some other days it only has a look at the homepage, or some more.

in this moment the freshbot is on his way around, I cannot see the deepbot at all.

3:59 am on Mar 11, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Jan 5, 2003
posts:202
votes: 0


I got hit by freshbot on the 8th and 9th.

Deepbot started showing around 6 EST today.

4:13 am on Mar 11, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 18, 2003
posts:128
votes: 0


Google is deep crawling my PR6 site right now...*216...didn't wait long after the update, eh? and I'm not ready for him..of course, I never am no matter how much I optimize all month....I always need mre time...yikes!
11:08 am on Mar 11, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 18, 2002
posts:145
votes: 0


Yes....my servers noticed DeepBot 216* 2 times(one at 1 am and one at 4 am italian time )

not ready at all.. :\

3:50 pm on Mar 11, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 2, 2002
posts:374
votes: 0


yeah, I see deepbot now too. Guess I was wrong.
Am I the only one who thinks that, given how late this update was and how late the crawl is we might not see the "March" Update (assuming we define the current update as the "February Update" as the previous was Januiary) until mid to late April? In other words maybe this is gonna be the rare 6-7 weeks between updates that happens once a year or so.
Otherwise, Google will have to be re-indexing on crawl results from March 10 to March 15(?) within 2 weeks or so (yeah, I know they have done that too occasionally - I think August was like that if I recall right).
4:55 pm on Mar 11, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 8, 2002
posts:223
votes: 0


I just changed some pages and put them up. Googlebot deepcrawled some of the pages on the site prior to my latest update and will obviously come back to take the rest. Will it notice that the pages it had cralwed before are now fresher and if it does will it crawl them again?

Thanks

4:56 pm on Mar 11, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 2, 2002
posts:446
votes: 0


Freshbot is busy on my site (PR6). Deepbot (216.239.*) has been "knocking" - ie, hit about 15 pages last night and this morning. That usually means it will start its full-scale deep crawl sometime today or tonight.

Deepbot is usually very polite and knocks first. :) Gives me a little extra time to get those last-minute fixes done.

5:42 pm on Mar 11, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member googleguy is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Oct 8, 2001
posts:2882
votes: 0


Hope everyone has been getting any new content up. :)
5:52 pm on Mar 11, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 24, 2002
posts:135
votes: 0


Not at my site yet. :(
6:25 pm on Mar 11, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 2, 2002
posts:374
votes: 0


I believe its been said on other threads but I wanted to say how much I appreciated both the update and deepcrawl headsup, GoogleGuy. Its nice to retain that feeling that we're all in this together.
For myself, I've been working on the assumption that it may be longer than average before the next update; if not, I'll be pleasantly surprised, if so I'll know I did everything possible to get my appropriate season pages in.
This 102 message thread spans 4 pages: 102