homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

Robot tracking script

 11:36 pm on Oct 20, 2004 (gmt 0)

Is it possible to create a tracking script to capture the following:

report shows which search engine robots retrieved which pages on which days

report to demonstrate potential problems, such as when particular pages are never indexed by any robot. With this information we can revaluate pages and optimize differently or can re-submit "problem" pages to specific search engines.



 11:52 pm on Oct 20, 2004 (gmt 0)

I have something that shows what a given robot has fetched today in alphebetical order, and furthermore gives counts of total files fetched, unique files fetched. Its a basic BASH script that uses a lot of the unix command line functions (ie it will work on a unix or linux server only) if your interested let me know.

[edit -added]
To clarify here analizes the logs, it is not a server side include or anything that looks at visitors real time.


 3:53 pm on Oct 21, 2004 (gmt 0)

I'd be interested, but need something for windows too....


 4:12 pm on Oct 21, 2004 (gmt 0)

If you look for "Webbot activity" you can find a few PHP scripts that you can adapt to your needs


 4:19 pm on Oct 21, 2004 (gmt 0)

Search for that in this forum and maybe Google too.

What is your main reasons for tracking the robots?


 12:40 am on Oct 24, 2004 (gmt 0)

I was in a session at SES NYC where the moderator recommended a product called Robot Manager for some of the things you mention. I haven't used it, but she said it helped their company get more information on spider activity.


Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved