homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

Robot.txt and logs

 6:03 pm on Jul 24, 2002 (gmt 0)

What does the file robot.txt do - I hear it but I don't know what it's about.

What are some of the things you guys look for in your log files. What software is good for that kind of thing or where do I look for them?

Web server logs?



 6:10 pm on Jul 24, 2002 (gmt 0)

1. A robot.txt file PROHIBITS nice spiders from spidering your website. (The naughty ones don't care...)

2. <...look for in log files..> I look for keywords and where my traffic is coming from...and adjust my pages accordingly.

3. Try www.analog.cx for free log analyzer.


 8:20 pm on Jul 24, 2002 (gmt 0)

Hi madcat,

Here is the definitive site for the robots.txt [robotstxt.org]

I like FastStats from Mach5 software for reading log info.


 1:27 am on Jul 26, 2002 (gmt 0)

Thanks for your replies, I will look over all of that...

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved