homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

How can I easily view all incomers?

10+ Year Member

Msg#: 521 posted 3:52 pm on Apr 3, 2001 (gmt 0)

I've seen many postings here on WmW describing this spider and that spider coming and going... But how can you really view all of this?

As of right now I've been using Webtrends complete analysis, but I've only seen ONE bot come in aside from the webtrends one to do the reporting. The one coming in is the Inktomi Slurp'er. The site has been up for quite a while and has been actively submitted to various engines...

Is there software I should look into purchasing to help monitor JUST the incoming spiders or will my Webtrends cover that?

Any help = MUCH appreciated!

~Happy Tuesday~



WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 521 posted 4:30 pm on Apr 3, 2001 (gmt 0)

Hi Eric, you are talking web trends log analyzer and not WebTrends live right? They make quite a few products. WebTrends Live is an online graphic counter that won't tell you anything about spiders - 75% useless to someone running a site.

The easiest thing to do if you are in doubt about your log analyzer, is to download the logs from your server, load them up in a editor and start looking for spider names, and calls to robots.txt.


10+ Year Member

Msg#: 521 posted 5:08 pm on Apr 3, 2001 (gmt 0)


Unfortunately, the "It-Guys" I don't think will let that fly. We have all of our client sites running off of a colocated box and only *they* have the ability to get in there and truly view the logs. I have requested that we use complete reporting through Webtrends Analyzer (not Live!) and it has been fairly adequate up until now.

Do you (or anyone viewing this...) know of any applications specifically tailored to SE Bot reporting?


WebmasterWorld Senior Member 10+ Year Member

Msg#: 521 posted 2:34 am on Apr 5, 2001 (gmt 0)

Hi Eric, I've been running around in circles with that issue for months now, and finally convinced one of our programmers to write a program.
I simply couldn't find this anywhere. It seems that most people that have this have developed "inhouse" tools.

The one our programmer wrote checks for:

date, time, site name, UA, IP address, and page name.

For now this is looking great. Does anyone know if this is enough, or if we should include something else into the program?


10+ Year Member

Msg#: 521 posted 1:55 pm on Apr 5, 2001 (gmt 0)


I know you're programmer really intended that to be distributed as a freeware application, right? :) Sorry, I couldn't resist. I'm glad to know though that you too had looked into this avenue and found nothing. I was feeling a tad on the discouraged side.

Thanks for the help and assistance!

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved