Forum Moderators: open
I can see somebody calling himself "SnoopRob/1.5" crawling through my pages.
Where can I see the results of this crawl, i.e. which SE is SnoopRob serving?
pikaia
some_ip - - [timestamp] "request protocol" status size "referrer" "user_agent"
usually you'll have the name of a machine in some_ip (the nslookup of the ip). status is usually 200 (success)
Please do not overrate statistics, they can only give hints to actual usage of your pages, [url="http://www.analog.cx/docs/webworks.html"]here[/url] is why ("how the web works" on analog homepage). If you don't have a logfile analyzer yet, analog (http://www.analog.cx) is a good start.
To asssociate some bot with an engine is essentially making educated guesses (eg. GoogleBot is serving google), largely based on the ip (if googlebot was serving google, it would have to come from an ip owned by google (which it does)).
And ofc, read through these forums, they're an excellent source of information.
I have the feeling that your posting does not quite fit my question, although the article "how the web works" is interesting.
It is not a matter of interpreting web server statistics - I am not at all interested in calculating how many visitors I have on my pages or anything like that.
It is just the fact that a user agent called SnoopRob very much behaved like a spider (getting dozens of linked pages in a couple of seconds).
Sometimes KIT-Fireball comes to see my pages, and a few days later I can check my pages in www.fireball.de - there they are. After Marvin having spidered my sites I check the results in www.northernlight.com. When I see GoogleBot crawling around, I check my ranking at google.com or google.de ...
I just wanted to know who sent SnoopRob on its way and where I can check the results of that crawl.
Thank you anyway.
