Freshbot reading most files twice

[NickH]

Freshbot (64.68.82.nnn) has been doing the rounds on my site for the last couple of days. I've noticed a strange thing that I've not seen before: it seems to be requesting most files twice. Sometimes the same file is requested within a few minutes; other times a few hours pass between the requests. Both requests appear to be successful.

Has anyone else seen this pattern? What can it mean?

[conor]

This is fairly normal: Most spiders will request pages more than once on a crawl, as they follow links through a site. This is accelearted if the bots are coming from seperate IP's.

[NickH]

Well, I've not seen Googlebot do this before. Previously, it has cleanly requested each page once.

Here is an excerpt from the log, illustrating what is happening. Doesn't this look like some kind of glitch? (Even though different IPs are requesting the same page?)

64.68.82.44 - - [16/Feb/2003:04:55:08 -0500] "GET /puzzles/p043h.html HTTP/1.0" 200 1374 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.78 - - [16/Feb/2003:04:58:44 -0500] "GET /puzzles/p043h.html HTTP/1.0" 200 1374 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.46 - - [16/Feb/2003:06:05:00 -0500] "GET /puzzles/p043s.html HTTP/1.0" 200 3402 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.69 - - [16/Feb/2003:06:10:44 -0500] "GET /puzzles/p026s.html HTTP/1.0" 200 4560 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.41 - - [16/Feb/2003:06:13:25 -0500] "GET /puzzles/p042s.html HTTP/1.0" 200 4396 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.52 - - [16/Feb/2003:06:15:25 -0500] "GET /puzzles/p027h.html HTTP/1.0" 200 1282 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.26 - - [16/Feb/2003:06:16:34 -0500] "GET /puzzles/p043s.html HTTP/1.0" 200 3402 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.56 - - [16/Feb/2003:06:19:27 -0500] "GET /puzzles/p027h.html HTTP/1.0" 200 1282 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.48 - - [16/Feb/2003:06:20:37 -0500] "GET /puzzles/p026s.html HTTP/1.0" 200 4560 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.43 - - [16/Feb/2003:06:25:01 -0500] "GET /puzzles/p041s.html HTTP/1.0" 200 5838 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.17 - - [16/Feb/2003:06:34:08 -0500] "GET /puzzles/p041s.html HTTP/1.0" 200 5838 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.46 - - [16/Feb/2003:06:38:30 -0500] "GET /puzzles/p041h.html HTTP/1.0" 200 1340 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.45 - - [16/Feb/2003:06:42:00 -0500] "GET /puzzles/p023a.html HTTP/1.0" 200 1286 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.37 - - [16/Feb/2003:06:46:24 -0500] "GET /puzzles/p041a.html HTTP/1.0" 200 1319 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.55 - - [16/Feb/2003:06:51:16 -0500] "GET /robots.txt HTTP/1.0" 200 32 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.55 - - [16/Feb/2003:06:51:17 -0500] "GET /puzzles/p044h.html HTTP/1.0" 200 1501 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.68 - - [16/Feb/2003:06:51:50 -0500] "GET /puzzles/p044s.html HTTP/1.0" 200 2968 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.52 - - [16/Feb/2003:06:57:51 -0500] "GET /puzzles/p044s.html HTTP/1.0" 200 2968 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.78 - - [16/Feb/2003:06:58:39 -0500] "GET /puzzles/p044h.html HTTP/1.0" 200 1501 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"