Welcome to WebmasterWorld Guest from

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Robot tracking script



11:36 pm on Oct 20, 2004 (gmt 0)

10+ Year Member

Is it possible to create a tracking script to capture the following:

report shows which search engine robots retrieved which pages on which days

report to demonstrate potential problems, such as when particular pages are never indexed by any robot. With this information we can revaluate pages and optimize differently or can re-submit "problem" pages to specific search engines.


11:52 pm on Oct 20, 2004 (gmt 0)

10+ Year Member

I have something that shows what a given robot has fetched today in alphebetical order, and furthermore gives counts of total files fetched, unique files fetched. Its a basic BASH script that uses a lot of the unix command line functions (ie it will work on a unix or linux server only) if your interested let me know.

[edit -added]
To clarify here analizes the logs, it is not a server side include or anything that looks at visitors real time.


3:53 pm on Oct 21, 2004 (gmt 0)

10+ Year Member

I'd be interested, but need something for windows too....


4:12 pm on Oct 21, 2004 (gmt 0)

10+ Year Member

If you look for "Webbot activity" you can find a few PHP scripts that you can adapt to your needs


4:19 pm on Oct 21, 2004 (gmt 0)

10+ Year Member

Search for that in this forum and maybe Google too.

What is your main reasons for tracking the robots?


12:40 am on Oct 24, 2004 (gmt 0)

10+ Year Member

I was in a session at SES NYC where the moderator recommended a product called Robot Manager for some of the things you mention. I haven't used it, but she said it helped their company get more information on spider activity.



Featured Threads

Hot Threads This Week

Hot Threads This Month