Freshbot & Robots.txt

almost 20 times in 6 hours

[DavidT]

Is it normal? Today it took robots.txt about 17 times in 6 hours and nothing else not even index.html. Has freshbot become some annoying pinging tool?

[leifwessman]

maby it's something wrong with your robots.txt file? it has happened to me... googlebot tried to grab it over and over and over.

[DavidT]

It had no trouble with it two days ago and i haven't changed it since then. My guess is that it's after some image files related to Froogle searches which are in disallowed folders and it just keeps coming back to make sure!?

[amoore]

It's been my experience that freshbot pulls the robots.txt file a lot. Here are some stats on freshbot pulling mine:

+--------+------+ 
¦ day ¦ hits ¦ 
+--------+------+ 
¦ Mar 15 ¦ 5 ¦ 
¦ Mar 16 ¦ 10 ¦ 
¦ Mar 17 ¦ 2 ¦ 
¦ Mar 18 ¦ 17 ¦ 
¦ Mar 19 ¦ 1 ¦ 
¦ Mar 20 ¦ 15 ¦ 
¦ Mar 21 ¦ 8 ¦ 
¦ Mar 22 ¦ 1 ¦ 
¦ Mar 23 ¦ 3 ¦ 
¦ Mar 24 ¦ 5 ¦ 
¦ Mar 25 ¦ 3 ¦ 
¦ Mar 26 ¦ 4 ¦ 
¦ Mar 27 ¦ 6 ¦ 
¦ Mar 28 ¦ 13 ¦ 
¦ Mar 29 ¦ 1 ¦ 
¦ Mar 30 ¦ 3 ¦ 
¦ Mar 31 ¦ 15 ¦ 
¦ Apr 1 ¦ 5 ¦ 
¦ Apr 2 ¦ 13 ¦ 
¦ Apr 3 ¦ 10 ¦ 
¦ Apr 5 ¦ 8 ¦ 
¦ Apr 6 ¦ 11 ¦ 
¦ Apr 7 ¦ 7 ¦ 
¦ Apr 8 ¦ 1 ¦ 
¦ Apr 9 ¦ 12 ¦ 
¦ Apr 10 ¦ 2 ¦ 
¦ Apr 11 ¦ 19 ¦ 
¦ Apr 12 ¦ 16 ¦ 
¦ Apr 13 ¦ 8 ¦ 
¦ Apr 14 ¦ 3 ¦ 
¦ Apr 15 ¦ 12 ¦ 
¦ Apr 16 ¦ 6 ¦ 
+--------+------+ 

I'll guess that each machine (and possibly each crawler instance on that machine) keeps its own copies of robots.txt, so it has to pull it more than once.

I wouldn't worry about it.