Welcome to WebmasterWorld Guest from 54.159.50.111

Forum Moderators: brotherhood of lan & mack

Message Too Old, No Replies

Robot.txt and logs

     
6:03 pm on Jul 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 15, 2002
posts:971
votes: 0


What does the file robot.txt do - I hear it but I don't know what it's about.

What are some of the things you guys look for in your log files. What software is good for that kind of thing or where do I look for them?

Web server logs?

6:10 pm on July 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 1, 2002
posts:1017
votes: 0


1. A robot.txt file PROHIBITS nice spiders from spidering your website. (The naughty ones don't care...)

2. <...look for in log files..> I look for keywords and where my traffic is coming from...and adjust my pages accordingly.

3. Try www.analog.cx for free log analyzer.

8:20 pm on July 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 23, 2000
posts:1277
votes: 0


Hi madcat,

Here is the definitive site for the robots.txt [robotstxt.org]

I like FastStats from Mach5 software for reading log info.

1:27 am on July 26, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 15, 2002
posts:971
votes: 0


Thanks for your replies, I will look over all of that...