homepage Welcome to WebmasterWorld Guest from 50.17.21.7
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Need help about File Robots.txt
how do I know spiders visit to my site or not?
viva




msg:1526438
 3:13 am on Nov 4, 2004 (gmt 0)

Dear all

I have a little wonder about how spiders visit to my site.

Do we have any tricks to notice that spiders are already visited to the site? and how to check how many time that spiders are visited to my site?

Thank you in advance.

 

Leosghost




msg:1526439
 7:29 am on Nov 4, 2004 (gmt 0)

Ask your host to implement a stats programme that shows spider visits separate from other visits ( you can of course "mine" your "raw" stats for this but if you are new to doing this then a simple pre sorted prog is better ..
For example "awstats" is included in virtually any site running "Cpanel" as the management GUI..
To implement awstats can be a PITA if you have to switch it all in manually ..but in 9 from 10 "Cpanels" it is in the "Master panel options" as a simple "click for on"...

This will give you the robot names as they visit ..the number of "hits" per visit etc ..

viva




msg:1526440
 10:15 am on Nov 4, 2004 (gmt 0)

Thank you very much and I will try it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved