Bot out walking

[Brett_Tabke]

Well, despite the false starts and confusing with the fresh bot, GoogleBot is out in no un certain terms today.

[northweb]

my sites are the same...no deep crawling.

[yoyo1]

This is like the 4th day that googlebot is crawling my new site. It's been going NONSTOP every 30 seconds or so! I wish it would stop soon.

[Paully]

Send her this way... I havent seen the deep crawler yet... PR5....

[dcheney]

deep crawl since last evening - about half done last I checked (PR5)

[shady]

Googlebot has made 1263 page requests at my site since 3rd Nov. This is definitely a "deep" crawl although many of the pages are new since the last google crawl, which may have some bearing on it. (Also PR5)

[echelon]

Monday - 41 Visits
Tuesday - 9 Visits
Wednesday - 53 Visits

I'm a fairly new site, PR4.. just indexed during last update.

I'm expecting more visits as Googlebot seems to be crawling my entire forum which I've converted into static .htm files just for her.

[echelon]

Make that 5 more for today... Seems like the crawler is visiting me 5 times/hours and grabbing the content one page at a time. Very friendly.. evenly paced.

[quotations]

My brand new site got deep crawled on the 4th and freshbot crawled on the 5th.

[wasmith]

Very good crawl for me, but its been spread out over the last few days, i believe everything has been hit once? its getting hard to keep track. I guess i need to write a script for the google deep crawl! So i know were i stand without manually going through to many log files.

[Hoople]

My one site was launched late July. It appeared in DMOZ in early August. GoogleBot deep crawled it in late August. Cached paged held aug 30th dates. Latest cached pages are now October 30th and November 2nd dated. I'm awaiting freshbot tags as it's been by the last 2 days :)

[echelon]

Update: Forget that 5 visits/hour thing I said. After I went to bed, Google deep crawled the rest of the site. Looks like it waited till off peak hours to do the bulk work.

[OldFatherTime]

My site seems to only attract the 'cautious' Google bots. Google grabs robots.txt which denies nothing, it then grabs the home page and disappears - frequently. Other bots take the whole site. One is busy crawling all of my images on a regular basis - but Google says "just the home page thanks.

Any ideas?

[yoyo1]

I'm still getting crawled by googlebot with my new site. It's now ruining my webalizer stats since everything is skewed. I wish it would stop already.

[warumauchnicht]

hi.

try to stop it with using your robots.txt. Disallow *Googlebot* -

[yoyo1]

But I don't want it to stop completely. I do want it to come back every month, just not every 30 seconds.

[starec]

You can configure your webalizer so that it ignores Googlebot hits completely.

If you have 3000+ spiderable pages you must be prepared for this kind of spidering. If your server can't handle it, limit the number of pages or (better) get a better machine/more bandwidth.

[taxpod]

Heck yesterday I was getting 1,500 pages an hour and I was so happy I couldn't keep the smile off my face. Sorry, but I don't think you have anything to complain about! If you can't figure out how to get Gbot ignored on your stats program, I suggest you learn to live with it once a month. IT IS A GOOD THING!