Reading log files

How!

[allybongo]

I recently managed to get my hands on my clients web logs in order to find out what spiders are visiting. The logs are produced every hour (bit of a nightmare!) but they all seem to look the same except for the IP address and the files accessed. How do I make sense of all this? I was expecting to find reference to googlebot etc.

This is an example of what every entry looks like (with IP and web site blanked). Do I have to note the IP address and hunt around to match it to a spider?

Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+5.0) http://www.example.com/
2002-08-05 13:05:01 111.222.33.44 - CLM 111.222.3.4 55 GET /images/thisisanimage.gif - 200 www.joeblogs.com

Its the first time I've had to do this so its all a mystery to me!

[Sinner_G]

You might want to check out this [draftlight.co.uk] page.

[allybongo]

Thanks for that! However, after ploughing my way through a few of the log files I finally found Scooter. It had identified itself so I am presuming that the others will aswell. Also asking the client to request a daily log so I dont have to plough through 24 every day!

Its so exciting! I shouted really loudly, "I've found a spider!" in the office and everyone screamed and started running for the door!

[Sinner_G]

Searchengineworld has a list of spiders and their IPs [searchengineworld.com] which could help you.