How can I easily view all incomers?

[Eric_Lander]

I've seen many postings here on WmW describing this spider and that spider coming and going... But how can you really view all of this?

As of right now I've been using Webtrends complete analysis, but I've only seen ONE bot come in aside from the webtrends one to do the reporting. The one coming in is the Inktomi Slurp'er. The site has been up for quite a while and has been actively submitted to various engines...

Is there software I should look into purchasing to help monitor JUST the incoming spiders or will my Webtrends cover that?

Any help = MUCH appreciated!

~Happy Tuesday~

[Brett_Tabke]

Hi Eric, you are talking web trends log analyzer and not WebTrends live right? They make quite a few products. WebTrends Live is an online graphic counter that won't tell you anything about spiders - 75% useless to someone running a site.

The easiest thing to do if you are in doubt about your log analyzer, is to download the logs from your server, load them up in a editor and start looking for spider names, and calls to robots.txt.

[Eric_Lander]

Brett,

Unfortunately, the "It-Guys" I don't think will let that fly. We have all of our client sites running off of a colocated box and only *they* have the ability to get in there and truly view the logs. I have requested that we use complete reporting through Webtrends Analyzer (not Live!) and it has been fairly adequate up until now.

Do you (or anyone viewing this...) know of any applications specifically tailored to SE Bot reporting?

[2_much]

Hi Eric, I've been running around in circles with that issue for months now, and finally convinced one of our programmers to write a program.
I simply couldn't find this anywhere. It seems that most people that have this have developed "inhouse" tools.

The one our programmer wrote checks for:

date, time, site name, UA, IP address, and page name.

For now this is looking great. Does anyone know if this is enough, or if we should include something else into the program?

[Eric_Lander]

2_much...

I know you're programmer really intended that to be distributed as a freeware application, right? :) Sorry, I couldn't resist. I'm glad to know though that you too had looked into this avenue and found nothing. I was feeling a tad on the discouraged side.

Thanks for the help and assistance!