Welcome to WebmasterWorld Guest from 54.162.96.103

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Robot tracking script

     
11:36 pm on Oct 20, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 6, 2004
posts:9
votes: 0


Is it possible to create a tracking script to capture the following:

∑ report shows which search engine robots retrieved which pages on which days

∑ report to demonstrate potential problems, such as when particular pages are never indexed by any robot. With this information we can revaluate pages and optimize differently or can re-submit "problem" pages to specific search engines.

11:52 pm on Oct 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 17, 2004
posts:174
votes: 0


I have something that shows what a given robot has fetched today in alphebetical order, and furthermore gives counts of total files fetched, unique files fetched. Its a basic BASH script that uses a lot of the unix command line functions (ie it will work on a unix or linux server only) if your interested let me know.

[edit -added]
To clarify here analizes the logs, it is not a server side include or anything that looks at visitors real time.

3:53 pm on Oct 21, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 6, 2004
posts:9
votes: 0


I'd be interested, but need something for windows too....
4:12 pm on Oct 21, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 29, 2003
posts:351
votes: 0


If you look for "Webbot activity" you can find a few PHP scripts that you can adapt to your needs
4:19 pm on Oct 21, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 6, 2004
posts:9
votes: 0


Search for that in this forum and maybe Google too.

What is your main reasons for tracking the robots?

12:40 am on Oct 24, 2004 (gmt 0)

New User

10+ Year Member

joined:Aug 11, 2004
posts:6
votes: 0


I was in a session at SES NYC where the moderator recommended a product called Robot Manager for some of the things you mention. I haven't used it, but she said it helped their company get more information on spider activity.

[softwaremirror.com...]

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members