How do you know when your web site is spidered?

Can tell you when your web site has been visited by a search engines

[brucec]

How do you know when? I always wondered.

[choster]

You need to check your access logs. Every time someone or something requests a document on your site, your web server logs the originating IP address and usually whatever the someone or something calls itself-- the "referrer string"-- as well as the status code (e.g. "200 OK" or "404 Not Found"), size, and other information about the visit.

For instance, a browser's referrer string might be "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)," whereas a search engine spider might announce itself as "Googlebot/2.1
(+http://www.googlebot.com/bot.html)." Of course, these headers can be spoofed, too.

The Tracking and Logging forum may be of interest to you [webmasterworld.com...] .

[edited by: choster at 7:33 pm (utc) on Jan. 26, 2004]

[agerhart]

Check your logs.

Then check this forum: [webmasterworld.com...]