Google Crawler

nonstop crawling

[wfernley]

I have been finding lately google has been crawling my site like mad. Already in august I have had 27000 hits from google which is much higher than the usual 1500 a month. My logs show this ip 66.249.65.80 is crawling my site nonstop.

Can anyone explain this, or has google changed their spyder lately?

Thanks :)

[goodroi]

There have been a few others reporting the same issue.

[wfernley]

What I find very odd is that its crawling pages I'm working on and there are no direct links to that page so how would the google crawler know about it?

For some reason I have a thought in the back of my head that its some sort of virus. :S

[theBear]

Does your web site leak file names because it generates a directory listing if the bot request ends with a / and there is no index entry in that directory?

[goodroi]

there are many ways they can find your urls, don't forget the most obvious - google toolbar

[theBear]

Yup toolbars will do it.

They like ET call home and blab.

[larryhatch]

I don't use toolbars. How does a toolbar find a new orphaned page (no links to page)? -Larry

[mahoogle]

I have two sites that have been getting thousands of googlebot requests daily. The majority are from the mozilla bot. They were never crawled like this in the past. Two things are interesting. One, it was crawled thousands of pages from one site, but only lists 500 pages as indexed. Secondly, the other pages has about 200K pages, but it lists 300K. These are probably unrelated.

One other important item is our site doesn't get much google traffic.

[theBear]

Larry,

I also do not run toolbars. However some of them phone home and tell home base what your browser has viewed.

So if you view a page that you are working on with a toolbar prior to even linking it to your site that info gets sent home.

I view all toolbars as spyware.

[larryhatch]

Thanks Bear: Exactly what I was afraid of, and a good reason not to use toolbars. -Larry

[wfernley]

Thanks for the info, although I am using Firefox and I dont have a toolbar loaded. Weird...