Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

How can I easily view all incomers?

3:52 pm on Apr 3, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 29, 2001
votes: 0

I've seen many postings here on WmW describing this spider and that spider coming and going... But how can you really view all of this?

As of right now I've been using Webtrends complete analysis, but I've only seen ONE bot come in aside from the webtrends one to do the reporting. The one coming in is the Inktomi Slurp'er. The site has been up for quite a while and has been actively submitted to various engines...

Is there software I should look into purchasing to help monitor JUST the incoming spiders or will my Webtrends cover that?

Any help = MUCH appreciated!

~Happy Tuesday~

4:30 pm on Apr 3, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 16

Hi Eric, you are talking web trends log analyzer and not WebTrends live right? They make quite a few products. WebTrends Live is an online graphic counter that won't tell you anything about spiders - 75% useless to someone running a site.

The easiest thing to do if you are in doubt about your log analyzer, is to download the logs from your server, load them up in a editor and start looking for spider names, and calls to robots.txt.

5:08 pm on Apr 3, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 29, 2001
votes: 0


Unfortunately, the "It-Guys" I don't think will let that fly. We have all of our client sites running off of a colocated box and only *they* have the ability to get in there and truly view the logs. I have requested that we use complete reporting through Webtrends Analyzer (not Live!) and it has been fairly adequate up until now.

Do you (or anyone viewing this...) know of any applications specifically tailored to SE Bot reporting?

2:34 am on Apr 5, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 12, 2000
votes: 0

Hi Eric, I've been running around in circles with that issue for months now, and finally convinced one of our programmers to write a program.
I simply couldn't find this anywhere. It seems that most people that have this have developed "inhouse" tools.

The one our programmer wrote checks for:

date, time, site name, UA, IP address, and page name.

For now this is looking great. Does anyone know if this is enough, or if we should include something else into the program?

1:55 pm on Apr 5, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 29, 2001
votes: 0


I know you're programmer really intended that to be distributed as a freeware application, right? :) Sorry, I couldn't resist. I'm glad to know though that you too had looked into this avenue and found nothing. I was feeling a tad on the discouraged side.

Thanks for the help and assistance!


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members