homepage Welcome to WebmasterWorld Guest from 107.20.34.144
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 47 message thread spans 2 pages: 47 ( [1] 2 > >     
Bot out walking
Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 6639 posted 12:00 pm on Nov 4, 2002 (gmt 0)

Well, despite the false starts and confusing with the fresh bot, GoogleBot is out in no un certain terms today.

 

clickclick

10+ Year Member



 
Msg#: 6639 posted 12:34 pm on Nov 4, 2002 (gmt 0)

Hi Brett

I take it that this is the main bot not the fresh bot?

216.239.46.121 NET: Google Inc. RDNS: crawl5.googlebot.com

Grumpus

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 6639 posted 12:48 pm on Nov 4, 2002 (gmt 0)

Correct, clickclick. That's the right one for the deep crawl. Actually, the '121' can be just about anything - they own the whole block (or close to it - I forget offhand and suck at keeping good notes).

Confirmed, though. She's walking.

G.

Rugles

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 6639 posted 1:21 pm on Nov 4, 2002 (gmt 0)

Yes I too can report some deep crawling from the 216 IP range. It has been a longer wait this time.

lazyz

10+ Year Member



 
Msg#: 6639 posted 1:49 pm on Nov 4, 2002 (gmt 0)

How can I tell the difference between the fresh bot and the deep crawling bot?

echelon

10+ Year Member



 
Msg#: 6639 posted 2:14 pm on Nov 4, 2002 (gmt 0)

Just wanted to confirm that I've spotted hits from:

216.239.46.101
216.239.46.140
216.239.46.77

kstprod

10+ Year Member



 
Msg#: 6639 posted 2:59 pm on Nov 4, 2002 (gmt 0)

I too, have finally been visited by Googlebot this morning! Although she only grabbed my robots.txt and index page, I'm still happier than ever that she visited me at all. Is it possible she will visit again during this crawl? Being that I just went live on 10/22, and this was her first visit, am I safe in assuming that I will most likely be included in the next update? I do have a few quality PR sites linking to me. And am I also safe in assuming that it should be somewhere around the end of this month? Just want to double check my conclusions, based on previous posts, so I'm not totally off track.

Sorry for all the questions, but as I'm sure you all can understand, I am new to Google and VERY excited to have seen her :)

Karen

jen24815

10+ Year Member



 
Msg#: 6639 posted 3:21 pm on Nov 4, 2002 (gmt 0)

Yep, me too.

Got all my pages; including the new ones in the subdir I was worried about. :)

Helpmebe1

10+ Year Member



 
Msg#: 6639 posted 3:40 pm on Nov 4, 2002 (gmt 0)

Kstprod,
Congrats! Yes, assuming you were deep crawled at the end of the month you should magically appear..

Go60Guy

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 6639 posted 4:03 pm on Nov 4, 2002 (gmt 0)

Just confirming that I'm seeing lots of 216s.

michaelbs

10+ Year Member



 
Msg#: 6639 posted 4:32 pm on Nov 4, 2002 (gmt 0)

Hi guys,
Can anyone tell me which is the best way I can tell if googlebot has visited my site? Is there a free stats tool that I can use for analyzing to see when she visits?

Thanks,
Mike

taxpod

10+ Year Member



 
Msg#: 6639 posted 4:39 pm on Nov 4, 2002 (gmt 0)

Definitely pulling down lots of pages. I've seen these fellows:

216.239.46.23
216.239.46.86
216.239.46.197
216.239.46.85
216.239.46.116
216.239.46.102

just to name the first couple. So I'd guess Brett is right about the entire block.

The thing I'm curious about is why so many IPs. Is this just the result of inbound links? What i mean is that if I go through my logs, I get a different IP for Googlebot on just about every line whereas usually with the freshbot, I get just one IP throughout the day.

gmoney

10+ Year Member



 
Msg#: 6639 posted 6:44 pm on Nov 4, 2002 (gmt 0)

michaelbs,

You can probably find what you are looking for in the “tracking and logging” forum:

[webmasterworld.com...]

Just browse through the titles until you see some relevant ones.

However, I can never seem to get the free ones I use (analog) to do exactly what I want so I often download my log files and write little programs of my own to pull out the information I want.

To track the Googlebot, I just extract every log entry with the word “googlebot” in it.

rfgdxm1

WebmasterWorld Senior Member rfgdxm1 us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 6639 posted 10:26 pm on Nov 4, 2002 (gmt 0)

And you doubted my word, Brett? Like I can't spot an IP in the logs starting with 216.*? :( ;) The deep crawler grabbed everything on both my sites. Not that this is a whole lot, but it insisted on having it all.

Finder

10+ Year Member



 
Msg#: 6639 posted 11:21 pm on Nov 4, 2002 (gmt 0)

Does the depth of Googlebot's indexing depend on your PR? For some reason Googlebot rarely goes deeper than the first set of links on my front page. Unfortunately this only brings up content indexes and very little content itself.

Googlebot filled up quite a chunk of the log file yesterday.

irock

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 6639 posted 11:20 pm on Nov 4, 2002 (gmt 0)

Googlebot just grabbed 1200 pages from my site in one night. IP start with 216.xxx..... Can anyone confirm?

Slade

10+ Year Member



 
Msg#: 6639 posted 12:05 am on Nov 5, 2002 (gmt 0)

Finder, if this is the case for you, you should deeplink to good content from your main pages.

See [searchengineworld.com...]

Finder

10+ Year Member



 
Msg#: 6639 posted 1:47 am on Nov 5, 2002 (gmt 0)

Thanks Slade.

I did that very thing last month after perusing WW for tips and tricks. I pulled up more links to deep content onto my front page. I think I need to do more though. I'll have to be a little creative. :)

bnhall

10+ Year Member



 
Msg#: 6639 posted 2:08 am on Nov 5, 2002 (gmt 0)

This is neat - I've got the deepcrawl bot AND the freshbot visiting at the same time. I guess this is why it's called a dance :)

taxpod

10+ Year Member



 
Msg#: 6639 posted 4:25 pm on Nov 5, 2002 (gmt 0)

Finder,

You've got a ton of parameters in your URL.

"?section=fantasy&sub=byauthor&auth=Robin+Wayne+Bailey"

is apparently just too much to get through. IMHO, shorten it up if you can or try one of the rewrite programs so that your parameters look like directories instead of parameters. I don't know anything about these but you need to do something to cut down the parameters.

Beachboy

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 6639 posted 4:57 pm on Nov 5, 2002 (gmt 0)

Yo Ms. Googlebot, walk on over here, got some new links for you to see, cold milk and chocolate chip cookies for you to munch on, too! Don't be shy, and I know you're hungry! :) :) :)

Finder

10+ Year Member



 
Msg#: 6639 posted 9:58 pm on Nov 5, 2002 (gmt 0)

taxpod,

That's the easiest way for me to do it. I could change "auth" to an ID number, but that's about it.

mortalfrog

10+ Year Member



 
Msg#: 6639 posted 10:40 pm on Nov 5, 2002 (gmt 0)

It might be the easiest way, but it's not going to deliver traffic...

I just completed a redesign for a dynamic site that now uses an ALA-type method to deliver static looking URL's.

Here's a really brief outline: each page in the database has a url like "blue-widget.html" in a url field. All requests for product pages are redirected to a php script that parses the URL and looks it up in the database to retrieve the product record and all necessary data.

Previously, this client encoded some of that data in the URL, like you do, and none of their pages were in google. As of the October update, they have 400 product pages in google, and the site is doing twice the business it did in October.

You can optimize your main pages all you want, but you're going to get far more traffic by having hundreds or thousands of pages in google than you ever will from just a dozen optimized html pages.

So, look up ways to make your URL's search engine friendly, and do it. Getting those pages in google is definitely worth it.

WebRookie

10+ Year Member



 
Msg#: 6639 posted 10:51 pm on Nov 5, 2002 (gmt 0)

Was visited by the freshbot first yesterday then deep crawl 216.239.46.124. Freshbot visits nearly every day.

jady

10+ Year Member



 
Msg#: 6639 posted 12:20 am on Nov 6, 2002 (gmt 0)

Glabbin' em all over here as well..

Finder

10+ Year Member



 
Msg#: 6639 posted 1:22 am on Nov 6, 2002 (gmt 0)

mortalfrog,

You are right. I analyzed the logs and it looks like Googlebot isn't taking any URL that has more than two variables in it. I'm working on eliminating some of the worst offenses. I'll have to wait until next month to see if it helps. I think it has all it wants from my site for this month.

Thanks everyone for the tips. It has helped a great deal!

EliteWeb

WebmasterWorld Senior Member eliteweb us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 6639 posted 1:23 am on Nov 6, 2002 (gmt 0)

Leeched robots.txt of new sites - will be a surefire next index i can feel it. All the sites will be up there somewhere :) And if not time will tell where they are. but we all know we love the #1-5 spots for ranking!

johnraphone

10+ Year Member



 
Msg#: 6639 posted 2:53 am on Nov 6, 2002 (gmt 0)

She's going nuts. When GoogleBot visits my sites, its a great day. :)

216.239.46.100, 216.239.46.101, 216.239.46.105, 216.239.46.124, 216.239.46.133, 216.239.46.140, 216.239.46.164, 216.239.46.166, 216.239.46.171, 216.239.46.172, 216.239.46.204, 216.239.46.220, 216.239.46.222, 216.239.46.223, 216.239.46.23, 216.239.46.236, 216.239.46.66, 216.239.46.82, 216.239.46.88

coosblues

10+ Year Member



 
Msg#: 6639 posted 3:07 am on Nov 6, 2002 (gmt 0)

She seems quite hungry to my delight. Last night grabbed my entire site - about a week before just came for a look. Keep on coming ms. google

creep

10+ Year Member



 
Msg#: 6639 posted 9:51 pm on Nov 6, 2002 (gmt 0)

Googlebot has has a different pattern this time around it seems. My sites are getting crawled very shallow. Noirmally they get a good deep one around this time. How is every one else doing regarding depth..?

This 47 message thread spans 2 pages: 47 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved