Googlebot

[Johny Favourite]

Hi people.

I've been monitoring the goolle bot on my site and I've noticed it keeps looking at incorrectbrowser.asp on my web server. This page was setup by the people that designed the site as a forward if the user had a really earlier version of ie. I'm not really that sure why it's still there actually.

Anyway, what my index.asp does is checks the version of ie and then if it's below something redirects to this page.

Is this going to be effecting how the bot spiders my site?

[Wail]

I can't tell for sure without looking at the code - but it seems most likely.

Here's my hunch as to what's going on.

Your site's code looks at the user agent of the visitor. The user agent is simply the technical name of the software being used to visit your site. Internet Explorer identifies itself as such, Netscape identifies itself, so does Opera, etc. Googlebot identifies itself as Googlebot.

I suspect your code is looking at the user agent of the visitor and using that to redirect them to the most appropriate version of your site for that browser or, at least, so code which is browser dependant can be fine tuned to suit that browser. This code hasn't any scope for anything which identifies itself as Googlebot. In fact having a special Googlebot version of your site or even Googlebot friendly fine tuning is against Google's Terms and Conditions.

So, in short, your site doesn't recognise Googlebot as a browser it knows and so throws the visiting robot off to incorrectbrowser.asp.

Take a look at Google's cache of your site. It might just be of incorrectbrowser.asp.

[Johny Favourite]

The thing is it seems to spider some of the site and then stop. So certain pages are spidered but no further.

[Wail]

Hiya,

Googlebot will crawl about 100k of a single page and it'll cache the same amount. If a page is 200k then don't expect to have SERPs for all of it.

On most crawls Google will look only at top level page (at least those that it considers top level based on inbound links). If you're seeing your top level pages and the pages just below them in the index but not pages that require three or more clicks to get to then don't be surprised if you don't see these in the index for a while/ever.

Of course, your top level pages might not have any sort of user agent detection on them.

Or it could be the menu system of your site. If the navigation of your site is user agent based then it'll only when Googlebot tries to navigate your site/explore into it that it'll be redirected.