homepage Welcome to WebmasterWorld Guest from 54.211.73.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Googlebot Spotted Deep Crawling
Update coming soon?
Net_Wizard



 
Msg#: 22439 posted 5:43 am on Mar 4, 2004 (gmt 0)

Active deep crawling activity early this morning March 3rd eastern time and been going on since this posting March 4 12:45 AM Eastern.

64.68.92.xx range Googlebot/2.1 (+http://www.googlebot.com/bot.html

I think this is it folks, everybody been waiting for :)

Perhaps a big update(including page rank) will follow.

Cheers

 

phantombookman

10+ Year Member



 
Msg#: 22439 posted 7:06 am on Mar 4, 2004 (gmt 0)

I don't pretend to know technically what that means but I hope the outcome is true.
Have put in loads of work thanks to what I have learnt on these boards in the past 2 months, still waiting to see the results.
Fingers crossed
Regards
Rod

pendanticist

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22439 posted 7:06 am on Mar 4, 2004 (gmt 0)

You mean this?

64.68.82.143 - - [01/Mar/2004:20:11:03 -0800] "GET /robots.txt HTTP/1.0" 200 1565 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.143 - - [01/Mar/2004:20:11:03 -0800] "GET / HTTP/1.0" 200 - "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Since the first of March I've had Twelve hits. Seven were for robots.txt, Four HTTP 1 and a lone 404 that'd been 301'd ages ago.

Unless you see something here that I don't.

Net_Wizard



 
Msg#: 22439 posted 8:01 pm on Mar 4, 2004 (gmt 0)

Sorry guys, I've been out all day long...trying to make a living ;)

Anyway...I mean this 64.68.92.xx 'the whole ip range'.
Just check it again as I posted...'ALL DAY LONG' March 4 with UA "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

and again going deep including URL's with '?'

I lost count by now but I got hundreds of Googlebot access.

It won't be long it would in you site.

markus007

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22439 posted 8:06 pm on Mar 4, 2004 (gmt 0)

i have thousands of googlebots on my site per day, but they just seem to be validating old URL's, no real deep crawling going on.

Net_Wizard



 
Msg#: 22439 posted 8:12 pm on Mar 4, 2004 (gmt 0)

I'm seeing some activity in some of my sites as well. Googlebot requesting some inner pages...nothing fast yet on my other sites about a few access per hour different pages and might escalate later on.

Am I the only one seeing this?

Net_Wizard



 
Msg#: 22439 posted 8:14 pm on Mar 4, 2004 (gmt 0)

thousands of googlebots on my site per day

Well, lucky you ;)

But for some of us who have been seeing only 1 Googlebot a day or not even one this might be something else.

PeterD

10+ Year Member



 
Msg#: 22439 posted 8:26 pm on Mar 4, 2004 (gmt 0)

May or may not be related: Anybody notice this new googlebot User-Agent string? I started seeing it March 2. Coming from legit googlebot IPs (crawler11 and crawler13):

"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The old string was

"Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

I also noted that the URL in the old string is being redirected to the URL in the new string.

(Forgive me if this has been mentioned before, but I couldn't find any mention of it.)

internetheaven

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22439 posted 9:32 pm on Mar 4, 2004 (gmt 0)

Yes, PeterD, someone has mentioned it a couple of days ago but no-one seemed to notice that the old URL now re-directs to the new one.
I've had a gut feeling that Google were doing an overhaul to their crawler since Florida. I figured that the low number of hits that everyone has experienced over the past month or so was due to testing of the new crawler. They wouldn't let a new crawler go as fast as the old one incase of bugs. Maybe testing is over and now the increase in hits is not an impending update (I certainly hope not anyway) but the new crawler being given more freedom to roam?

Trisha

10+ Year Member



 
Msg#: 22439 posted 9:48 pm on Mar 4, 2004 (gmt 0)

Something seems to be going on, finally. I've had a new section of pages added to a site for a few weeks, and googlebot wouldn't crawl them. After reading this thread, I checked again, and now almost all of the pages have white bars!

internetheaven

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 22439 posted 10:31 pm on Mar 4, 2004 (gmt 0)

Does anyone know of a Google Dance tool that still works? All of the ones I used to check with are now useless due to the IP changes.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved