How can I tell when I have been spider has visited my site?
Is it only by my server

The information you'd need to evaluate are in your access_log files. Do you have access to them? I only ask because I understand individual host servers handle this differently.

Pendanticist.

Check with your site hosting service.

If it's a freebie host don't expect to have access to the raw logs, where you can see a record of every visitor that comes to your site, including spiders.

If it's a paid hosting service, you probably have access to the raw logs via an FTP client like WS_FTP which is available for free on the web.

I have raw access logs, I can open them and save them to my computer but it saves it as a .gzip Which program do I use to open and view my logs? I've been trying to find this out for some time now. Anyone know how I can do this?

To open the .gz file ftp it to your computer, then open it with Winzip (free trial versions available on the web). It takes a little hacking to learn how to use Winzip to expand the file, though. I associate the .gz file with Winzip, then do a right mouse click on the filename in Windows Explorer, select Winzip and click on 'expand to here'. Not a very good explanation, is it? So get some instructions for Winzip and read them ... probably beats my insistance on hacking rather than reading by a long shot.

ExtremeExports,

What follows is only a small part of what mayor is coaching you on.

---------------------------------------------------------------------------------

I don't know if your host server is the same as mine, but when you're looking at those files from within your FTP client - the very first one (at the top of the list) should be viewable as a .txt file even after you download it to your hard drive. Just that first one, mind you.

That file shows the most current activity and is also the data that shows in your daily stats page when you view it.

Once the time period set by your host server (usually a week) runs out, that viewable .txt file gets converted by your host server into one of those .gz files and a new .txt file is started.

As you become better at evaluating access_log files you might want to keep tabs on that one as it shows traffic more incrimentally and not enmasse like the .gz files.

Myself, I download that one several times a day.

Just a thought.

Pendanticist.

Powerarchiver available at powerarchiver.com is freeware and requires no hacking. However, most log analyzers will read packed logfiles.

Andreas

Once I have access to my log files; how do I determine the IP addresses that
belong to spiders?

Thank you everyone for the help. I will try it and see if I understood the instructions on how to open it.
Extreme

There is a terrific resource for spiders here. [searchengineworld.com] Look at the IP addresses from that page and it will indicate what you should look for in your site logs.

You will find further information on spiders here in this forum. Use the WebmasterWorld site search to identify newer or "rogue" spiders.

When you look in your logs, generally, the respectable spiders look for your robots.txt file first. That should help you see the spider visits.

I've downloaded the rawaccess log to my computer, extracted it with winzip and then tried to open it but it gives me the following error: Program to big to fit into memory.
What do I do now? :-(

Reboot

Use a better editor (have a look at the editors forum for some suggestions) or get more memory.

It's trying to be opened in MSDOS. I am so frustrated right now, that I am unable to do something that sounds so simple. :(

okay! I finally am able to view it in a text editor and it looks very confusing. But thank you everyone again. :)

Perhaps this info on Apache´s logfile [httpd.apache.org] will help.

THANK YOU ANDREASFRIEDRICH!
I figured it out! Now I get it! :o

--Petra

You´re welcome Petra.

Happy log checking ;)

Andreas