Forum Moderators: open

Message Too Old, No Replies

Typical spider activity

beginner's question

         

Mr_Busby

2:03 am on Jul 28, 2003 (gmt 0)

10+ Year Member



I've just installed the AXS tracking system which seems to be working great.

Might be a stupid question but how do you tell if it's a spider looking at your pages, and not just a person?

Now that I can track my visitors, what kind of activity would indicate that a hit is a spider?

wilderness

3:26 am on Jul 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



what kind of activity would indicate that a hit is a spider?

IP, UA, the pattern in which the bot grags your pages, the speed your pages are grabbed (depending upon individual content,) most bots show no interest in images, the pages the bot views based on the referrer (if possible.)

I find solitary page views with no images and HEAD requests to be most troubling. So much in fact that I make a notation of the visit.

There are likely a few more which have not come to mind immediately.

In the end, only the webmaster is familiar with his/her own content and that webmaster learns from monitoring their own traffic flow and making a determination based on their acummulated experience.

You haven't specified the inquiry for compliant or non-compliant bots?
The compliant bots, most everybody desires and their compliancy or activity is generally NOT an issue.

The non-compliant bots are the PITA's. Most of which are not inclined to ID themselves. Or do ID themsleves as another.