Welcome to WebmasterWorld Guest from 220.127.116.11
As of right now I've been using Webtrends complete analysis, but I've only seen ONE bot come in aside from the webtrends one to do the reporting. The one coming in is the Inktomi Slurp'er. The site has been up for quite a while and has been actively submitted to various engines...
Is there software I should look into purchasing to help monitor JUST the incoming spiders or will my Webtrends cover that?
Any help = MUCH appreciated!
The easiest thing to do if you are in doubt about your log analyzer, is to download the logs from your server, load them up in a editor and start looking for spider names, and calls to robots.txt.
Unfortunately, the "It-Guys" I don't think will let that fly. We have all of our client sites running off of a colocated box and only *they* have the ability to get in there and truly view the logs. I have requested that we use complete reporting through Webtrends Analyzer (not Live!) and it has been fairly adequate up until now.
Do you (or anyone viewing this...) know of any applications specifically tailored to SE Bot reporting?
The one our programmer wrote checks for:
date, time, site name, UA, IP address, and page name.
For now this is looking great. Does anyone know if this is enough, or if we should include something else into the program?
I know you're programmer really intended that to be distributed as a freeware application, right? :) Sorry, I couldn't resist. I'm glad to know though that you too had looked into this avenue and found nothing. I was feeling a tad on the discouraged side.
Thanks for the help and assistance!