Robot.txt and logs

[madcat]

What does the file robot.txt do - I hear it but I don't know what it's about.

What are some of the things you guys look for in your log files. What software is good for that kind of thing or where do I look for them?

Web server logs?

[ScottM]

1. A robot.txt file PROHIBITS nice spiders from spidering your website. (The naughty ones don't care...)

2. <...look for in log files..> I look for keywords and where my traffic is coming from...and adjust my pages accordingly.

3. Try www.analog.cx for free log analyzer.

[JamesR]

Hi madcat,

Here is the definitive site for the robots.txt [robotstxt.org]

I like FastStats from Mach5 software for reading log info.

[madcat]

Thanks for your replies, I will look over all of that...