Welcome to WebmasterWorld Guest from 54.166.232.243

Forum Moderators: open

Message Too Old, No Replies

Bot out walking

     
12:00 pm on Nov 4, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38074
votes: 17


Well, despite the false starts and confusing with the fresh bot, GoogleBot is out in no un certain terms today.
12:34 pm on Nov 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:July 23, 2002
posts:53
votes: 0


Hi Brett

I take it that this is the main bot not the fresh bot?

216.239.46.121 NET: Google Inc. RDNS: crawl5.googlebot.com

12:48 pm on Nov 4, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 13, 2002
posts:676
votes: 0


Correct, clickclick. That's the right one for the deep crawl. Actually, the '121' can be just about anything - they own the whole block (or close to it - I forget offhand and suck at keeping good notes).

Confirmed, though. She's walking.

G.

1:21 pm on Nov 4, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2001
posts:2145
votes: 0


Yes I too can report some deep crawling from the 216 IP range. It has been a longer wait this time.
1:49 pm on Nov 4, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Jan 28, 2002
posts:240
votes: 0


How can I tell the difference between the fresh bot and the deep crawling bot?
2:14 pm on Nov 4, 2002 (gmt 0)

New User

10+ Year Member

joined:Oct 29, 2002
posts:19
votes: 0


Just wanted to confirm that I've spotted hits from:

216.239.46.101
216.239.46.140
216.239.46.77

2:59 pm on Nov 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 24, 2002
posts:147
votes: 0


I too, have finally been visited by Googlebot this morning! Although she only grabbed my robots.txt and index page, I'm still happier than ever that she visited me at all. Is it possible she will visit again during this crawl? Being that I just went live on 10/22, and this was her first visit, am I safe in assuming that I will most likely be included in the next update? I do have a few quality PR sites linking to me. And am I also safe in assuming that it should be somewhere around the end of this month? Just want to double check my conclusions, based on previous posts, so I'm not totally off track.

Sorry for all the questions, but as I'm sure you all can understand, I am new to Google and VERY excited to have seen her :)

Karen

3:21 pm on Nov 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 11, 2002
posts:137
votes: 0


Yep, me too.

Got all my pages; including the new ones in the subdir I was worried about. :)

3:40 pm on Nov 4, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 9, 2002
posts:422
votes: 0


Kstprod,
Congrats! Yes, assuming you were deep crawled at the end of the month you should magically appear..
4:03 pm on Nov 4, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 8, 2002
posts:993
votes: 0


Just confirming that I'm seeing lots of 216s.
4:32 pm on Nov 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 3, 2002
posts:163
votes: 0


Hi guys,
Can anyone tell me which is the best way I can tell if googlebot has visited my site? Is there a free stats tool that I can use for analyzing to see when she visits?

Thanks,
Mike

4:39 pm on Nov 4, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:May 1, 2002
posts:439
votes: 0


Definitely pulling down lots of pages. I've seen these fellows:

216.239.46.23
216.239.46.86
216.239.46.197
216.239.46.85
216.239.46.116
216.239.46.102

just to name the first couple. So I'd guess Brett is right about the entire block.

The thing I'm curious about is why so many IPs. Is this just the result of inbound links? What i mean is that if I go through my logs, I get a different IP for Googlebot on just about every line whereas usually with the freshbot, I get just one IP throughout the day.

6:44 pm on Nov 4, 2002 (gmt 0)

Full Member

10+ Year Member

joined:May 16, 2002
posts:223
votes: 0


michaelbs,

You can probably find what you are looking for in the “tracking and logging” forum:

[webmasterworld.com...]

Just browse through the titles until you see some relevant ones.

However, I can never seem to get the free ones I use (analog) to do exactly what I want so I often download my log files and write little programs of my own to pull out the information I want.

To track the Googlebot, I just extract every log entry with the word “googlebot” in it.

10:26 pm on Nov 4, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member rfgdxm1 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 12, 2002
posts:4479
votes: 0


And you doubted my word, Brett? Like I can't spot an IP in the logs starting with 216.*? :( ;) The deep crawler grabbed everything on both my sites. Not that this is a whole lot, but it insisted on having it all.
11:21 pm on Nov 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 18, 2002
posts:131
votes: 0


Does the depth of Googlebot's indexing depend on your PR? For some reason Googlebot rarely goes deeper than the first set of links on my front page. Unfortunately this only brings up content indexes and very little content itself.

Googlebot filled up quite a chunk of the log file yesterday.

11:20 pm on Nov 4, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 2, 2002
posts:1099
votes: 0


Googlebot just grabbed 1200 pages from my site in one night. IP start with 216.xxx..... Can anyone confirm?
12:05 am on Nov 5, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:July 16, 2001
posts:545
votes: 0


Finder, if this is the case for you, you should deeplink to good content from your main pages.

See [searchengineworld.com...]

1:47 am on Nov 5, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 18, 2002
posts:131
votes: 0


Thanks Slade.

I did that very thing last month after perusing WW for tips and tricks. I pulled up more links to deep content onto my front page. I think I need to do more though. I'll have to be a little creative. :)

2:08 am on Nov 5, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 15, 2002
posts:520
votes: 0


This is neat - I've got the deepcrawl bot AND the freshbot visiting at the same time. I guess this is why it's called a dance :)
4:25 pm on Nov 5, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:May 1, 2002
posts:439
votes: 0


Finder,

You've got a ton of parameters in your URL.

"?section=fantasy&sub=byauthor&auth=Robin+Wayne+Bailey"

is apparently just too much to get through. IMHO, shorten it up if you can or try one of the rewrite programs so that your parameters look like directories instead of parameters. I don't know anything about these but you need to do something to cut down the parameters.

4:57 pm on Nov 5, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 4, 2002
posts:1068
votes: 0


Yo Ms. Googlebot, walk on over here, got some new links for you to see, cold milk and chocolate chip cookies for you to munch on, too! Don't be shy, and I know you're hungry! :) :) :)
9:58 pm on Nov 5, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 18, 2002
posts:131
votes: 0


taxpod,

That's the easiest way for me to do it. I could change "auth" to an ID number, but that's about it.

10:40 pm on Nov 5, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:June 20, 2002
posts:60
votes: 0


It might be the easiest way, but it's not going to deliver traffic...

I just completed a redesign for a dynamic site that now uses an ALA-type method to deliver static looking URL's.

Here's a really brief outline: each page in the database has a url like "blue-widget.html" in a url field. All requests for product pages are redirected to a php script that parses the URL and looks it up in the database to retrieve the product record and all necessary data.

Previously, this client encoded some of that data in the URL, like you do, and none of their pages were in google. As of the October update, they have 400 product pages in google, and the site is doing twice the business it did in October.

You can optimize your main pages all you want, but you're going to get far more traffic by having hundreds or thousands of pages in google than you ever will from just a dozen optimized html pages.

So, look up ways to make your URL's search engine friendly, and do it. Getting those pages in google is definitely worth it.

10:51 pm on Nov 5, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 12, 2000
posts:395
votes: 0


Was visited by the freshbot first yesterday then deep crawl 216.239.46.124. Freshbot visits nearly every day.
12:20 am on Nov 6, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 9, 2002
posts:426
votes: 0


Glabbin' em all over here as well..
1:22 am on Nov 6, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 18, 2002
posts:131
votes: 0


mortalfrog,

You are right. I analyzed the logs and it looks like Googlebot isn't taking any URL that has more than two variables in it. I'm working on eliminating some of the worst offenses. I'll have to wait until next month to see if it helps. I think it has all it wants from my site for this month.

Thanks everyone for the tips. It has helped a great deal!

1:23 am on Nov 6, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 5, 2001
posts:2729
votes: 8


Leeched robots.txt of new sites - will be a surefire next index i can feel it. All the sites will be up there somewhere :) And if not time will tell where they are. but we all know we love the #1-5 spots for ranking!
2:53 am on Nov 6, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 22, 2002
posts:65
votes: 0


She's going nuts. When GoogleBot visits my sites, its a great day. :)

216.239.46.100, 216.239.46.101, 216.239.46.105, 216.239.46.124, 216.239.46.133, 216.239.46.140, 216.239.46.164, 216.239.46.166, 216.239.46.171, 216.239.46.172, 216.239.46.204, 216.239.46.220, 216.239.46.222, 216.239.46.223, 216.239.46.23, 216.239.46.236, 216.239.46.66, 216.239.46.82, 216.239.46.88

3:07 am on Nov 6, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Oct 19, 2002
posts:308
votes: 0


She seems quite hungry to my delight. Last night grabbed my entire site - about a week before just came for a look. Keep on coming ms. google
9:51 pm on Nov 6, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:July 26, 2001
posts:139
votes: 0


Googlebot has has a different pattern this time around it seems. My sites are getting crawled very shallow. Noirmally they get a good deep one around this time. How is every one else doing regarding depth..?
This 47 message thread spans 2 pages: 47
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members