| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Gold Sponsor 2015! |
|Robot.txt and logs|
What does the file robot.txt do - I hear it but I don't know what it's about.
What are some of the things you guys look for in your log files. What software is good for that kind of thing or where do I look for them?
Web server logs?
1. A robot.txt file PROHIBITS nice spiders from spidering your website. (The naughty ones don't care...)
2. <...look for in log files..> I look for keywords and where my traffic is coming from...and adjust my pages accordingly.
3. Try www.analog.cx for free log analyzer.
Here is the definitive site for the robots.txt [robotstxt.org]
I like FastStats from Mach5 software for reading log info.
Thanks for your replies, I will look over all of that...
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved