Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Need help about File Robots.txt

how do I know spiders visit to my site or not?



3:13 am on Nov 4, 2004 (gmt 0)

10+ Year Member

Dear all

I have a little wonder about how spiders visit to my site.

Do we have any tricks to notice that spiders are already visited to the site? and how to check how many time that spiders are visited to my site?

Thank you in advance.


7:29 am on Nov 4, 2004 (gmt 0)

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Ask your host to implement a stats programme that shows spider visits separate from other visits ( you can of course "mine" your "raw" stats for this but if you are new to doing this then a simple pre sorted prog is better ..
For example "awstats" is included in virtually any site running "Cpanel" as the management GUI..
To implement awstats can be a PITA if you have to switch it all in manually ..but in 9 from 10 "Cpanels" it is in the "Master panel options" as a simple "click for on"...

This will give you the robot names as they visit ..the number of "hits" per visit etc ..


10:15 am on Nov 4, 2004 (gmt 0)

10+ Year Member

Thank you very much and I will try it.

Featured Threads

Hot Threads This Week

Hot Threads This Month