Spiders on the default page

is it common for them to act like this?

[Gmorgan]

Hi all

When looking through my weblogs I can see that often spiders visit my site and the only page they visit is the default/home page, just looking at that over and over again.

Is it normal for your logs to look something like:

defaul.asp
defaul.asp
defaul.asp
defaul.asp
defaul.asp
defaul.asp
defaul.asp
defaul.asp (and so on another 20 times)

after a spider has visited?

If this shouldn't be happening then what would be the reason it is?

Thanks, GM

[Nuttakorn]

I need to study the log file at date & time which the spider has been crawled. IMO, if you have update your content in homepage or rest of your homepage, the spider will crawl that page many time than the page without update.

[Sanenet]

Maybe your robots.txt is banning everything except the homepage?

Or do you have the meta tag "robots" = "nofollow" in your homepage?

[Gmorgan]

Ok, I've just asked the more technical guys here (i'm not one of them so this is mostly foreign to me) and:

- There's been no changes to the homepage or the rest of the site for a while
- There is no robotx.txt file

Now i'm even more stumped

[Sanenet]

Well, it's default.htm and robots.txt, but I assume thats a typo :)

There could be server setting problem, has anybody been playing around with the server settings recently?

Are the links from the homepage normal?

Can you post the appropiate logfile lines (inline with TOS), se we can see exactly what is happening?

[mincklerstraat]

I had a similar problem with one of my sites, some @#*% thing was requesting the same pages, a zillion times in short succession. Turned out that 'thing' was me after looking at the logs more carefully - this was back when I predominantly used windows, and had some sort of scumware somewhere on the computer. Cleaned the computer of scumware and it stopped happening. I can only feel sorry for all the sites I'd visited in the meantime, and all the extra bandwidth I'd cost them.