Forum Moderators: DixonJones

Message Too Old, No Replies

How can you see a crawler has visited your site?

         

glenn7

9:02 am on Sep 30, 2003 (gmt 0)

10+ Year Member



Hi,

I guess it should be obvious that I'm new to this

I'm a graphic designer who originally thought he could just 'flash' the web, kinda came back on that opinion after some research... :)!

and it seems the more I research, the more I find and the more answers AND questions I get...

but this one is a simple question (so simple everybody knows it and i can't find it...)

How can you see a crawler has visited your page? I've read several posts in here talking about logs and showing lines of code...
I figured a service like webstat or similar can provide me with this, but hey, I'm still wandering what 'raw acces files' are, so...

thnx for any help, already had my eyes opened the few hours I spent here ;)

Sinner_G

9:24 am on Sep 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



originally thought he could just 'flash' the web, kinda came back on that opinion after some research

That's a very good start ;). And welcome to WebmasterWorld.

How can you see a crawler has visited your page?

Most web statistics packages will have a subsection (e.g. WebTrends) called spiders or similar, where you will find that information.

what 'raw acces files' are

Basically, the web server your files are logs every request made to it by a browser. Say you have a html page including 2 images, the web server will registrate 3 requests. These are logged into a text file, often placed on your web server in a folder called logs (or similar). The exact information you will find there depends on what system/web server you are using.

glenn7

1:06 pm on Sep 30, 2003 (gmt 0)

10+ Year Member



Thanx, one step closer now... :)

antirack

4:49 am on Oct 2, 2003 (gmt 0)

10+ Year Member



We are using Urchin, which is one of the best commercial packages around. As far as I know there is no function to see spider "usage" or statistics but I would be very interested to know some more about spiders visiting our site (no. of pages requested, data transferred, etc) in order to see who and what is going on.