Monitoring Spiders

[glenv]

I am always hesitant to post a question since it shows my ignorance :-)

What software do you recommend to monitor search engine crawls? I have been busy optimizing and submitting my sites and would love to feel the excitement you all have when you alert us that the engines are crawling....

I have multiple sites. My host/server has software traffic monitoring but not so good...

[KakenBetaal]

Thanks for the heads-up, Brett. I've not yet seen the Google deepcrawler, but then I have only a PR5 site. I'm sure she'll be here by tomorrow. :)

Good question, Glenv, that's one I'd like to know about as well. I tend to just watch my raw logs and keep an eye on sessions on my forums every now and then, but I'm sure there's a better way out there.

[swizz]

Hello glenv,

I'm using live stats from deepmatrix.com, this stat server has an option to watch for some ips.. i.e, Spiders IPs.
Altavista Crawler Watching IP: 216.39.48.*
Google Crawler Watching IP: 216.239.*
Alexa Crawler Watching IP: 209.237.238.*
Google Freshbot Watching IP: 64.68.*

Another software I found to check my stats is "Log Analyzer" from bitstrike software (cheaper as live stats), you can even see how many times a spider comes by

Cheers
Steven

[GetVisible]

I use the free openwebscope - http://www.openwebscope.com/

any others out there?

GetVisibleUK

[hooloovoo22]

strange...anyone else seeing tabkebot out there as well?

[wingslevel]

av and inktomi are really working me over...

no googlebot for me yet on a pr6 site :(

[Chris_D]

"...anyone else seeing tabkebot out there as well?"

ROTFL!

I thought I saw a tabkebot crawling from a bar the other night - but it was someone else

:)

[bluecorr]

does that software show what pages are being pulled?

[hooloovoo22]

might be a stretch, but if you use coldfusion i have a pretty good googlebot watch script. stickymail me if interested

[wilderness]

The traffic stats that most hosts provide are good tools in most instances. The problem lay in that the softwares are configured in a stanardized format by the host. Generally this results in excessive stats on cosmetic images which are useless. They also are pre-configured to limit the top few of most stat references.

Some time ago I scourred the internet for freeware stats software.
One very useful software was named "Logalizer" however the number of lines it could read varied by your machines processor and memory. I've not tried it recently perhaps a newer version exists.

The BEST I found is named "Analog" and although it seems a task (at first) to configure it is entirely configureable and returns quality stats which YOU desire. With no limitations that I've seen so far. An excellent tool.

[KakenBetaal]

Analog is superb traffic analysis software, even more so given that it's free. I'm not so sure on using it to monitor spiders, though, but perhaps I haven't configured it correctly.

ATM I do a find on my raw logfile and dump all the googlebot referrers to a separate text file. That way I can browse through the visits and see the deep/fresh crawler and what they've been looking at when.

[markatapd]

Hi I am a interested in this analog software. Does anyone know the URL that it may be obtained from and does anyone else know of any other good FREE tracking software. I am just a lttle guy trying to start his own business and cannot afford a lot of expense at this time.

[wilderness]

Searches at google
Logalizer

[google.netscape.com...]

Analog
[google.netscape.com...]

Similar pages
[google.netscape.com...]

Glad I'm not your customer ;-)

[markatapd]

thanks for the help - hey I'm getting better :-)