Welcome to WebmasterWorld Guest from 54.147.134.218

Forum Moderators: open

Message Too Old, No Replies

Googlebot Spotted Deep Crawling

Update coming soon?

     
5:43 am on Mar 4, 2004 (gmt 0)

Preferred Member

joined:May 28, 2002
posts:377
votes: 0


Active deep crawling activity early this morning March 3rd eastern time and been going on since this posting March 4 12:45 AM Eastern.

64.68.92.xx range Googlebot/2.1 (+http://www.googlebot.com/bot.html

I think this is it folks, everybody been waiting for :)

Perhaps a big update(including page rank) will follow.

Cheers

7:06 am on Mar 4, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 30, 2003
posts:625
votes: 0


I don't pretend to know technically what that means but I hope the outcome is true.
Have put in loads of work thanks to what I have learnt on these boards in the past 2 months, still waiting to see the results.
Fingers crossed
Regards
Rod
7:06 am on Mar 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


You mean this?

64.68.82.143 - - [01/Mar/2004:20:11:03 -0800] "GET /robots.txt HTTP/1.0" 200 1565 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.143 - - [01/Mar/2004:20:11:03 -0800] "GET / HTTP/1.0" 200 - "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Since the first of March I've had Twelve hits. Seven were for robots.txt, Four HTTP 1 and a lone 404 that'd been 301'd ages ago.

Unless you see something here that I don't.

8:01 pm on Mar 4, 2004 (gmt 0)

Preferred Member

joined:May 28, 2002
posts:377
votes: 0


Sorry guys, I've been out all day long...trying to make a living ;)

Anyway...I mean this 64.68.92.xx 'the whole ip range'.
Just check it again as I posted...'ALL DAY LONG' March 4 with UA "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

and again going deep including URL's with '?'

I lost count by now but I got hundreds of Googlebot access.

It won't be long it would in you site.

8:06 pm on Mar 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 24, 2003
posts:814
votes: 0


i have thousands of googlebots on my site per day, but they just seem to be validating old URL's, no real deep crawling going on.
8:12 pm on Mar 4, 2004 (gmt 0)

Preferred Member

joined:May 28, 2002
posts:377
votes: 0


I'm seeing some activity in some of my sites as well. Googlebot requesting some inner pages...nothing fast yet on my other sites about a few access per hour different pages and might escalate later on.

Am I the only one seeing this?

8:14 pm on Mar 4, 2004 (gmt 0)

Preferred Member

joined:May 28, 2002
posts:377
votes: 0


thousands of googlebots on my site per day

Well, lucky you ;)

But for some of us who have been seeing only 1 Googlebot a day or not even one this might be something else.

8:26 pm on Mar 4, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 29, 2002
posts:64
votes: 0


May or may not be related: Anybody notice this new googlebot User-Agent string? I started seeing it March 2. Coming from legit googlebot IPs (crawler11 and crawler13):

"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The old string was

"Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

I also noted that the URL in the old string is being redirected to the URL in the new string.

(Forgive me if this has been mentioned before, but I couldn't find any mention of it.)

9:32 pm on Mar 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 25, 2003
posts:2527
votes: 0


Yes, PeterD, someone has mentioned it a couple of days ago but no-one seemed to notice that the old URL now re-directs to the new one.
I've had a gut feeling that Google were doing an overhaul to their crawler since Florida. I figured that the low number of hits that everyone has experienced over the past month or so was due to testing of the new crawler. They wouldn't let a new crawler go as fast as the old one incase of bugs. Maybe testing is over and now the increase in hits is not an impending update (I certainly hope not anyway) but the new crawler being given more freedom to roam?
9:48 pm on Mar 4, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 12, 2002
posts:565
votes: 0


Something seems to be going on, finally. I've had a new section of pages added to a site for a few weeks, and googlebot wouldn't crawl them. After reading this thread, I checked again, and now almost all of the pages have white bars!
10:31 pm on Mar 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 25, 2003
posts:2527
votes: 0


Does anyone know of a Google Dance tool that still works? All of the ones I used to check with are now useless due to the IP changes.