starting out with log file analysis

beginners guide needed

[skav]

I want to start monitoring log files and have invested in Robot-Manager. I understand the basics of how spiders work etc but am having a lot of difficulty setting the program up to give me useful info. Can anyone direct me to some useful background 'help' resources. Specifically I need a list of the most important spider IP addresses. For some reason Robot-Manager isn't identifying the spiders by name only IP address.

Many thanks

[hasbeen]

Welcome, Skav (it it hasn't been offered yet)..

Use the site search function for spider IP addresses. There are a ton floating around here. You may also want to check here:
Search Engine Spider Identification [webmasterworld.com]

[volatilegx]

I have posted lists of spider ips here [iplists.com].

[skav]

Thank you very much, both of you.

[Crazy_Fool]

volatilegx - is your list as up to date as possible with the entries from here as well ?

[volatilegx]

Crazy_Fool

Well it's pretty close. Nobody's perfect :) The lists were built from the perspective of a cloaker... so some engines that serve cached pages may be left out or commented out.

[Crazy_Fool]

thanks volatilegx - knowing that someone is maintaining a list is encouraging - i'll dig out some more soon.

[volatilegx]

Fantomaster also keeps an updated list, but his costs money :)