Welcome to WebmasterWorld Guest from 54.166.87.123

Forum Moderators: brotherhood of lan & mack

Message Too Old, No Replies

Robot.txt and logs

   
6:03 pm on Jul 24, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



What does the file robot.txt do - I hear it but I don't know what it's about.

What are some of the things you guys look for in your log files. What software is good for that kind of thing or where do I look for them?

Web server logs?

6:10 pm on Jul 24, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



1. A robot.txt file PROHIBITS nice spiders from spidering your website. (The naughty ones don't care...)

2. <...look for in log files..> I look for keywords and where my traffic is coming from...and adjust my pages accordingly.

3. Try www.analog.cx for free log analyzer.

8:20 pm on Jul 24, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi madcat,

Here is the definitive site for the robots.txt [robotstxt.org]

I like FastStats from Mach5 software for reading log info.

1:27 am on Jul 26, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for your replies, I will look over all of that...