homepage Welcome to WebmasterWorld Guest from 54.226.180.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Robot tracking script
melncali




msg:892024
 11:36 pm on Oct 20, 2004 (gmt 0)

Is it possible to create a tracking script to capture the following:

report shows which search engine robots retrieved which pages on which days

report to demonstrate potential problems, such as when particular pages are never indexed by any robot. With this information we can revaluate pages and optimize differently or can re-submit "problem" pages to specific search engines.

 

nalin




msg:892025
 11:52 pm on Oct 20, 2004 (gmt 0)

I have something that shows what a given robot has fetched today in alphebetical order, and furthermore gives counts of total files fetched, unique files fetched. Its a basic BASH script that uses a lot of the unix command line functions (ie it will work on a unix or linux server only) if your interested let me know.

[edit -added]
To clarify here analizes the logs, it is not a server side include or anything that looks at visitors real time.

melncali




msg:892026
 3:53 pm on Oct 21, 2004 (gmt 0)

I'd be interested, but need something for windows too....

DanA




msg:892027
 4:12 pm on Oct 21, 2004 (gmt 0)

If you look for "Webbot activity" you can find a few PHP scripts that you can adapt to your needs

melncali




msg:892028
 4:19 pm on Oct 21, 2004 (gmt 0)

Search for that in this forum and maybe Google too.

What is your main reasons for tracking the robots?

glenn




msg:892029
 12:40 am on Oct 24, 2004 (gmt 0)

I was in a session at SES NYC where the moderator recommended a product called Robot Manager for some of the things you mention. I haven't used it, but she said it helped their company get more information on spider activity.

[softwaremirror.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved