Forum Moderators: phranque

Message Too Old, No Replies

Robots/spiders indexing my sites

How many index other sites?

         

Sunnydale

7:37 pm on Sep 9, 2005 (gmt 0)

10+ Year Member



Looking at my Cpanel today I was wondering if the following robots were the norm in this day an age for the amount of spiders/robots that should be indexing my sites which have been on the net for 2+ years.
This is what I'm getting (without the other stats [i.e plus 123 + 31] after each of these robots) on my stats on the Cpanel.

Inktomi Slurp
Googlebot
MSNBot
Unknown robot (identified by 'crawl')
Unknown robot (identified by hit on 'robots.txt')
Uknown robot (identified by 'spider')
AskJeeves
Alexa (IA Archiver)
Unknown robot (identified by 'robot')

I've submitted my sites properly many times without over submitting & was wondering if there are any robots I'm missing out on which would maybe do my site any favours, if so how can I reach them?

JAB Creations

2:12 am on Sep 10, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thats old school awstats right there...

It doesn't really detect spiders real well unless you personally develope the libraries. I'm reworking my version before I go public again (security hole) but yeah, if you play with awstats and mod the libraries a little and look in to things you can really see whats going on. :-)

Sunnydale

10:54 am on Sep 10, 2005 (gmt 0)

10+ Year Member



I wasn't aware I could do anything with awstats, how can I modify them and libraries? to find out whats really going on? Does this mean other hosting providers provide better stats or is it something I need to be doing?