Welcome to WebmasterWorld Guest from 23.22.206.103

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Robot tracking script

     
11:36 pm on Oct 20, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 6, 2004
posts:9
votes: 0


Is it possible to create a tracking script to capture the following:

report shows which search engine robots retrieved which pages on which days

report to demonstrate potential problems, such as when particular pages are never indexed by any robot. With this information we can revaluate pages and optimize differently or can re-submit "problem" pages to specific search engines.

11:52 pm on Oct 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 17, 2004
posts:174
votes: 0


I have something that shows what a given robot has fetched today in alphebetical order, and furthermore gives counts of total files fetched, unique files fetched. Its a basic BASH script that uses a lot of the unix command line functions (ie it will work on a unix or linux server only) if your interested let me know.

[edit -added]
To clarify here analizes the logs, it is not a server side include or anything that looks at visitors real time.

3:53 pm on Oct 21, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 6, 2004
posts:9
votes: 0


I'd be interested, but need something for windows too....
4:12 pm on Oct 21, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 29, 2003
posts:351
votes: 0


If you look for "Webbot activity" you can find a few PHP scripts that you can adapt to your needs
4:19 pm on Oct 21, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 6, 2004
posts:9
votes: 0


Search for that in this forum and maybe Google too.

What is your main reasons for tracking the robots?

12:40 am on Oct 24, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 11, 2004
posts:6
votes: 0


I was in a session at SES NYC where the moderator recommended a product called Robot Manager for some of the things you mention. I haven't used it, but she said it helped their company get more information on spider activity.

[softwaremirror.com...]