Normal spider pattern

what is it?

[grnidone]

I saw this post [webmasterworld.com]and had to wonder

What is considered a "normal" spidering pattern? It seems like google checks the robots.txt file and then the index page several times before it spiders the rest of the site.

What is normal? What is preferred?

-G

[littleman]

What is normal? What is preferred?
Two very tough questions. I would say in general there is no 'normal' pattern for spiders. 25 in 8 seconds would be a bit on the extreme end -- heading close to a DOS status. A nice spider would come in first to the robots.txt, and then gently crawl the site, it could not request more than one page every couple of minutes, and do it during non-peek hours.

[Woz]

>and do it during non-peek hours.

In which time zone? Is it possible to determine location and therefor timezone of a hosting computer?

Onya
Woz

[grnidone]

Kicking this to the top...I'd like to see if someone can answer Woz's question.
-G

[ggrot]

Dont all internet protocols send time as a time and offset from GMT? It does in emails at least.

[startup]

G,
Try this new toy:
[searchengineworld.com...]

By the look of it, time and date are available but, location is not.

[Woz]

>By the look of it, time and date are available but, location is not.

OK, slightly OT, but how then does a Search Engine that specifies the submitted site must be located within a certain area know whether it is or not? I had this happen the other day when I submited one of my sites with 100% Australian content to an Australian SE who immediately rejected 'cos it was not hosted in Aus. How'd they know that?

Onya
Woz

[startup]

My guess would be, IP range of the DNS.