How To Tell Which SE's Visited Which Pages

[MrSpeed]

Seems like people post things like "Googlebot spidered 40 of my pages and even pagexyz.html on 3/11/03"

I have never seen log analysis programs that I am familiar with able to get this granular(webtrends, webalizer).

The way I have attempted to do it is to download my raw log files and do a search in a text editor. Whic is ok for small log files but there has to be a better way.

So how do people determine which pages google spidered and on what date?

[sugarkane]

Personally I use 'grep' on the raw log files from a Linux command line. It's basically equivalent to searching in a text editor, but generally more flexible and can cope with large files rather better.

I've heard good things about a shareware program called 'Wingrep' which gives the same functionality on Windows machines.

[WebRankInfo]

Instead of analysing your logs, you may track the bots in real time. As soon as you detect a bot's visit, you save it (name, date, etc.) in order to compute statistics. If you're interested, I have developped such a program and offer it as an open source free application.

[Hagstrom]

> I've heard good things about a shareware program called 'Wingrep' which gives the same functionality on Windows machines.

There are plenty of freeware versions of grep and sed available for Windows.

[Receptional Andy]

>>If you're interested, I have developed such a program and offer it as an open source free application

I've been looking for something that does this. What are you using to track the bot visits - SSI or something similar?

[sugarkane]

> freeware versions

That's even better :)

[WebRankInfo]

"even better" than "open source free application"? ;-)
For each robot, I choose whether to base the detection on IP address or on User Agent.
I'm using PHP (server side script)

[Receptional Andy]

Have you put this on the web somewhere? If so can you sticky me the URL or some info please WebRankInfo?